Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Is there a way to hide URLs to prevent source from being exposed? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Is there a way to hide URLs to prevent source from being exposed?

2»

Comments

  • jmgcaguiclajmgcaguicla Member
    edited May 2021

    @notarobo said:
    what about the award

    What do you mean? he said it right there, it's the highest form of honor/award @yoursunny can give anyone.

    @yoursunny said:
    I'll record another push-up video in your honor, within a month.

    Thanked by 1yoursunny
  • jarjar Patron Provider, Top Host, Veteran

    @sgno1 said:

    @jar said:
    Reverse proxy. Server 1 pulls it from server 2 and delivers to client, client has no idea server 2 exists.

    I'm curious how this is better than the SSHFS/Rclone method?

    More consistently reliable, less weird.

  • sgno1sgno1 Member

    @jar said:

    @sgno1 said:

    @jar said:
    Reverse proxy. Server 1 pulls it from server 2 and delivers to client, client has no idea server 2 exists.

    I'm curious how this is better than the SSHFS/Rclone method?

    More consistently reliable, less weird.

    Reliable how?

  • WilliamWilliam Member
    edited May 2021

    @sgno1 said: Reliable how?

    rclone has some quirks, it is not usable for large scale data (lots of random IO). "Endpoint not connected" is a common issue, so you need scripts to check and remount etc. etc.

    I only use it as it is definitely the best option for Google Apps/Drive, aside of google driuve filestream which is WIndows/OSX only.

    Thanked by 1jar
  • jarjar Patron Provider, Top Host, Veteran
    edited May 2021

    @sgno1 said:

    @jar said:

    @sgno1 said:

    @jar said:
    Reverse proxy. Server 1 pulls it from server 2 and delivers to client, client has no idea server 2 exists.

    I'm curious how this is better than the SSHFS/Rclone method?

    More consistently reliable, less weird.

    Reliable how?

    Tricking an OS into thinking a remote store is a local one will include all kinds of quirks when something doesn’t go as planned. Monitoring for various conditions being required long term for sure. I have scripts that attempt to write to remote stores every 5 minutes just to make sure they’re up, because I can’t rely on the OS to report a brief failure in a consistent way.

    It’s sort of like building your own imperfect tool for the job, it’s weird because there’s already one made for the role, the reverse proxy.

  • wpyogawpyoga Member

    I'm reading @FAT32 's code and I'm thinking: it's just calling youtube-dl to parse the video info and download the best quality video and audio streams. Then it calls ffmpeg to combine the two streams. Am I missing something ...? On the inspector, I'm only seeing video data being sent over websocket.

    (I'm not an expert on this kind of stuff, and I barely know JS)

    Thanked by 2yoursunny FAT32
  • FAT32FAT32 Administrator, Deal Compiler Extraordinaire
    edited May 2021

    @yoursunny said:
    You are really good!
    Come to our next NDN hackathon, maybe?

    I'll record another push-up video in your honor, within a month.

    Thanks, honestly I think this is an interesting technology but I only know very little out of it, let alone understanding how to publish a new object to the network.


    @wpyoga said:
    I'm reading @FAT32 's code and I'm thinking: it's just calling youtube-dl to parse the video info and download the best quality video and audio streams. Then it calls ffmpeg to combine the two streams. Am I missing something ...? On the inspector, I'm only seeing video data being sent over websocket.

    (I'm not an expert on this kind of stuff, and I barely know JS)

    Ya that's more or less about it, the thing here is understand / guess how it was encoded / format of the packets at the first place. I also thought about just read from the same WebSocket used in Yoursunny's pushup page and call it a day but that itself have more variables such as adaptive streaming (So it might mix the video in different resolutions)

    Truth to be told, I actually have no idea about NDN in the past so well it is also a learning experience for me. There's lots of code in NDNts and sadly I don't see much love / stars in those repos, @yoursunny is working hard to write them.

    I first read the code from this 2 repos:

    And get to understand that the video is encoded using Shaka Packager as DASH format, but the video can be both vp9 (webm) / h264 (mp4).

    Then I take a look at this repo:

    To get an understand about what are the packages that we can use, I actually missed out @ndn/cat at the beginning that could make the process even simpler.

    Anyway, the idea of using youtube-dl is I am not able to find a good enough node.js library that parse DASH file and download the streams. So I decided to make use of the very powerful youtube-dl that can do the magic for me.

    youtube-dl is already sufficient but to merge the audio/video I used ffmpeg to do the job.

  • Daniel15Daniel15 Veteran

    @errhead said: I've been doing something like that using straight SSHFS, sounds like mergerFS will greatly improve performance

    SSHFS is really not designed to be high performance. It's a very poor choice for large files or when you need multiple concurrent transfers. It doesn't have any concurrency (meaning only one transfer can be happening at a time), no block access (even if you just want a small part of a file, the entire file needs to be transferred across the network), and no locking. Try NFS instead, or for read-only access you might have better luck with a HTTP proxy (backend server serves the files using HTTP, then some other server proxies them).

  • jmgcaguiclajmgcaguicla Member
    edited May 2021

    @FAT32 said:
    Ya that's more or less about it, the thing here is understand / guess how it was encoded / format of the packets at the first place. I also thought about just read from the same WebSocket used in Yoursunny's pushup page and call it a day but that itself have more variables such as adaptive streaming (So it might mix the video in different resolutions)

    Good stuff.

    I just found the usage for the library kind of weird, the fact the face variable didn't get used right after initialization threw me off. But then I saw the call to fetch not taking in any parameters which lead me to believe it's using a singleton/global default connection/factory instance that got initialized during the connectToTestbed call.

    Normally, you'd have handles to all the stuff you need to make a call. Just kind of a weird pattern, I haven't read much of the library but I guess you could also manually manage that state if you wanted to.

    Thanked by 2yoursunny lentro
  • FAT32FAT32 Administrator, Deal Compiler Extraordinaire
    edited May 2021

    @jmgcaguicla said:
    I just found the usage for the library kind of weird, the fact the face variable didn't get used right after initialization threw me off. But then I saw the call to fetch not taking in any parameters which lead me to believe it's using a singleton/global default connection/factory instance that got initialized during the connectToTestbed call.

    Normally, you'd have handles to all the stuff you need to make a call. Just kind of a weird pattern, I haven't read much of the library but I guess you could also manually manage that state if you wanted to.

    100% agree, this is what confused me at the beginning also, I was wondering where else can I use the face? Why does 1 library affect another? It drives me crazy at first :joy:

    But I have to say the TypeScript / JSDoc is pretty well documented.

  • sgno1sgno1 Member

    @Daniel15 said:

    @errhead said: I've been doing something like that using straight SSHFS, sounds like mergerFS will greatly improve performance

    SSHFS is really not designed to be high performance. It's a very poor choice for large files or when you need multiple concurrent transfers. It doesn't have any concurrency (meaning only one transfer can be happening at a time), no block access (even if you just want a small part of a file, the entire file needs to be transferred across the network), and no locking. Try NFS instead, or for read-only access you might have better luck with a HTTP proxy (backend server serves the files using HTTP, then some other server proxies them).

    I see, I mean I haven't really had any issue with SSHFS video serving so far. It currently mounts a local mount to the server and then uses read streams in Node to serve these files. Btw important note, I'm only reading files I won't be writing anything. I did try NFS with read-only but people advised me in LET to not use it remotely because it is not designed to be used publicly, and to only use it within a private/LAN network. I'm not too sure about the HTTP proxy, I ideally want a solution like my existing one (SSHFS Local Mount + Node.js ReadStream Serve from Local Mount).

  • jmgcaguiclajmgcaguicla Member
    edited May 2021

    @Daniel15 said:
    no block access (even if you just want a small part of a file, the entire file needs to be transferred across the network)

    What? No, SFTP handles partial file operations just fine.

  • sgno1sgno1 Member

    @jmgcaguicla said:

    @Daniel15 said:
    no block access (even if you just want a small part of a file, the entire file needs to be transferred across the network)

    What? No, SFTP handles partial file operations just fine.

    Oh really? Since SSHFS is based on SFTP, I assume even partial file operations are fine 🤔

  • Daniel15Daniel15 Veteran
    edited May 2021

    @sgno1 said: I did try NFS with read-only but people advised me in LET to not use it remotely because it is not designed to be used publicly, and to only use it within a private/LAN network

    That's right. If you're accessing a remote server, you can use it over a VPN (eg. WireGuard). https://d.sb/2020/12/nfs-howto. NFS lets you use CacheFS too, to get read caching on your client (if you want to use it).

    @jmgcaguicla said: What? No, SFTP handles partial file operations just fine.

    Oh yeah, it's very possible you're correct, but I'm not sure whether SSHFS fully takes advantage of that. I remember reading on ServerFault that there were some limitations of SSHFS block access (I think it was that SSHFS will always cache the entire file even if you only request a portion of it), but I can't find the post now.

    Do note that SSHFS is not actively maintained at the moment:

    SSHFS is shipped by all major Linux distributions and has been in production use across a wide range of systems for many years. However, at present SSHFS does not have any active, regular contributors, and there are a number of known issues (see the bugtracker). The current maintainer continues to apply pull requests and makes regular releases, but unfortunately has no capacity to do any development beyond addressing high-impact issues. When reporting bugs, please understand that unless you are including a pull request or are reporting a critical issue, you will probably not get a response.

    https://github.com/libfuse/sshfs

  • errheaderrhead Member

    SSHFS was a useful for testing the concept. I was suprised by how effective it was in my low volume usage. But yeah, suboptimal long term. After a day of googling and copy pasting, I've updated one of my test PeerTube servers to use rclone as a service to map the drive instead. Seems to add a bunch more options for the off-site storage as well.
    Added MergerFS to merge the local recently added folder and remote back catalog folder which should take care of most of the needed caching automatically if the cron prune job works right. Saving money and learning effective tools, what a wonderful site this is ;)

  • yoursunnyyoursunny Member, IPv6 Advocate

    @yoursunny said:

    @FAT32 said:

    @yoursunny said:
    First person that successfully downloads a video (must be the actual file or packets; screen recording doesn't count) will get an award.

    Done.

    You are really good!
    Come to our next NDN hackathon, maybe?

    I'll record another push-up video in your honor, within a month.

    Award delivered.
    https://pushups.ndn.today select 2021-06-03 title

    Moreover, if the "uncaught promise rejection" refers to a UDP RECVMSG error, it's been fixed in latest nightly.
    I never noticed this error because it doesn't occur on Windows.


    @FAT32 said:

    @jmgcaguicla said:
    I just found the usage for the library kind of weird, the fact the face variable didn't get used right after initialization threw me off. But then I saw the call to fetch not taking in any parameters which lead me to believe it's using a singleton/global default connection/factory instance that got initialized during the connectToTestbed call.

    Normally, you'd have handles to all the stuff you need to make a call. Just kind of a weird pattern, I haven't read much of the library but I guess you could also manually manage that state if you wanted to.

    100% agree, this is what confused me at the beginning also, I was wondering where else can I use the face? Why does 1 library affect another? It drives me crazy at first :joy:

    But I have to say the TypeScript / JSDoc is pretty well documented.

    There's a global Forwarder instance in the @ndn/fw package.
    Many functions can accept Forwarder as an option, with the default being that global instance.

    Thanked by 2FAT32 bulbasaur
  • FAT32FAT32 Administrator, Deal Compiler Extraordinaire

    @yoursunny said:
    Award delivered.
    https://pushups.ndn.today select 2021-06-03 title

    Moreover, if the "uncaught promise rejection" refers to a UDP RECVMSG error, it's been fixed in latest nightly.
    I never noticed this error because it doesn't occur on Windows.

    I don't know why but it feels so weird to watch someone doing pushups :joy: Thanks for the award :)

    By the way I noticed that your push up skill has improved a lot! Shorter interval / resting time and more accurate posture. (I am still very weak on pushups, can only done less than 10)

    Thanks for fixing the bug, honestly I wonder why you are so dedicated on working on NDN.

    Thanked by 1yoursunny
  • sgno1sgno1 Member

    @FAT32 said:

    @yoursunny said:
    Award delivered.
    https://pushups.ndn.today select 2021-06-03 title

    Moreover, if the "uncaught promise rejection" refers to a UDP RECVMSG error, it's been fixed in latest nightly.
    I never noticed this error because it doesn't occur on Windows.

    I don't know why but it feels so weird to watch someone doing pushups :joy: Thanks for the award :)

    By the way I noticed that your push up skill has improved a lot! Shorter interval / resting time and more accurate posture. (I am still very weak on pushups, can only done less than 10)

    Thanks for fixing the bug, honestly I wonder why you are so dedicated on working on NDN.

    FAT32 is my role model.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @FAT32 said:

    @yoursunny said:
    Award delivered.
    https://pushups.ndn.today select 2021-06-03 title

    I don't know why but it feels so weird to watch someone doing pushups :joy: Thanks for the award :)

    Learn from @lentro .
    My push-ups are his favorite pastime.

    I noticed that your push up skill has improved a lot! Shorter interval / resting time and more accurate posture.

    If you are comparing to the 2020-12-24 title, one reason is, that video was the fourth take.

    I do not carry a tripod.
    To record an outdoor video, I need to place the phone between tree branches or rocks.
    Problems occur frequently:

    • The phone fell from where it's placed.
    • The audio is mushed because the microphone is underwater.
    • The lighting is too bright or too dark.
    • I am out of the frame.

    At the fourth take, I've already done 96 push-ups, so that the form is worse.

    Indoor videos are easier because I'm familiar with the chairs, and there's no wind to knock off the phone.
    Most only need one take.

    (I am still very weak on pushups, can only done less than 10)

    Several providers have push-up discounts:

    • RackNerd backpack with 15 push-ups
    • Nexril $1 off per push-up, up to 24, from full price
    • NexusBytes 26.9% off with 269 push-ups
    • NexusBytes 200% off with 1000 push-ups

    You'd start training.

    honestly I wonder why you are so dedicated on working on NDN.

    I have a PhD on Named Data Networking.

    Thanked by 3Daniel15 lentro FAT32
  • wotetiwoteti Member

    I skipped the whole thread so someone may have mentioned this:

    Encode your url with jwt/paseto. That also means that you have to template your html, so they can't be static. Easy enough to do with php/python/nodejs/etc.

  • sgno1sgno1 Member
    edited June 2021

    @seriesn 200% recurring discount for each month that we do 1000 push-ups?

    Thanked by 1yoursunny
  • @sgno1 said:
    @seriesn 200% recurring discount for each month that we do 1000 push-ups?

    In one day yes.

  • @sgno1 said:
    Is there a cheap way to hide URLs to prevent the source from being exposed? I have a few videos that I would like to share, and I'm searching for a cheap solution. One could be using PHP to pass the video as a header but through the networking tab the source can still be found.

    Is there any alternative solution for this (any programming language works)? I could download the videos from the link and serve them that way but my server is just not fast enough to do that for each new video that would be requested.

    Any help, suggestions or tips would be appreciated!

    F11.

    Thanked by 3yoursunny _MS_ sgno1
Sign In or Register to comment.