We use cookies and other tools to enhance your experience on our website and to analyze our web traffic.
For more information about these cookies and the data collected, please refer to our Privacy Policy.

How to download full datasets

2 posts
Was this reply useful? Learn more...
 
[-]
robertthomas +1 point · over 9 years ago

What is the process to download a full dataset (e.g., all CHAT edf's)

Robert Thomas

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +1 point · over 9 years ago

Hi Robert,

We provide an official download tool called nsrr which is a Ruby gem.

Installation instructions for the gem are here: https://github.com/nsrr/nsrr-gem

In short, once you have Ruby installed, you can do the following two steps:

Install nsrr Ruby Gem

gem install nsrr --no-document

Finally, to download only the EDFs for chat, you can type:

nsrr download chat/polysomnography/edfs

The downloader supports pause and resume, so that it will pick up where it last left off.

NSRR Gem Console Output

The above is a screenshot of the app running through the CHAT folder

Additionally, each page in a dataset's download folder contains the specific nsrr command needed to download that particular folder once you have signed in and filled out a Data Access and Use Agreement for that particular dataset.

2 posts
Was this reply useful? Learn more...
 
[-]
robertthomas +0 points · over 9 years ago

Looks like I need a download token. Robert

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · over 9 years ago

After signing in, you can access your download token here: https://sleepdata.org/token

Additionally, the download token will only work for datasets for which you have an approved DAUA.

2 posts
Was this reply useful? Learn more...
 
[-]
zhuwang +0 points · about 9 years ago

hello , i followed your instructions as you said,and i type the website in the bowser,which returned the download token. however ,i could type the download token in the console, the only thing i can do is to press the enter button.unfortunately, it can create the folder but any contens.what should i do?

2 posts
Was this reply useful? Learn more...
 
[-]
zhuwang +0 points · about 9 years ago

sorry , i could not type the download token in the console.....

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · about 9 years ago

Hi Zhu,

The easiest way is to copy and paste the download token from the website directly into the console when using the NSRR gem. Make sure to include the number before the dash - as well as the rest of the string. The NSRR gem hides the input on the screen, but will still register the token if entered correctly. The gem still allows you to download publicly available files (like forms and data dictionaries) without entering a password which is why you sometimes see the folder structure generated without any files in it. A future version of the gem (v0.2.0+) should provide better feedback if the password was entered correctly or not. Hope that helps and thanks for the feedback!

For others interested in using the NSRR gem for downloading large numbers of files easily, you can view a short tutorial here:

https://sleepdata.org/demo#download-data

11 posts
Was this reply useful? Learn more...
 
[-]
stephanyfulda +0 points · almost 9 years ago

Dear NSRR team,

I just got my DAUA approved and a mere 20 minutes later I already downloaded the first data set with the nsrr Ruby Gem without any problems. So, thanks very much for the absolutely clear and easy to follow tutorial! The only question I would like to ask is whether there is any way to specify a different download path? Thanks very much!

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · almost 9 years ago

Dear Stephany,

Glad to hear that you were able to use the nsrr gem!

If you want to only download a portion of a dataset, you can do so by navigating to the appropriate subfolder under the dataset files.

For example, if you only want to download datasets (and not EDFs) for SHHS, you can go to the datasets subfolder. Near the bottom of the page you will see a command listed that will only download that subfolder, in this case nsrr download shhs/datasets.

There are also other parameters you can set that you can read about here on our NSRR gem Documentation page.

If you need any further questions, I'd be happy to clarify further, thanks!

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +1 point · almost 9 years ago

Just as another quick reference, here are a couple of nsrr gem commands to download different datasets:

SHHS Files

  • nsrr download shhs - Download ALL files for SHHS
  • nsrr download shhs/polysomnography/edfs - Download EDFs for SHHS
  • nsrr download shhs/datasets - Download all datasets for SHHS

CHAT Files

  • nsrr download chat - Download ALL files for CHAT
  • nsrr download chat/polysomnography/edfs - Download EDFs for CHAT
  • nsrr download chat/datasets - Download all datasets for CHAT

Also remember to run the command in the same location on your computer, that way the nsrr gem can resume if you already downloaded files from a dataset.

If your current directory in the command prompt or terminal is "~/Desktop" for example, running nsrr download chat/polysomnography/edfs will create a folder in "~/Desktop/chat/polysomnography/edfs". If you then run the full CHAT download command (nsrr download chat) from the "~/Desktop", then it would already see that it has downloaded the EDFs, and continue with "~/Desktop/chat/datasets" and other subfolders.

11 posts
Was this reply useful? Learn more...
 
[-]
stephanyfulda +0 points · almost 9 years ago

Dear Remo,

got it! As you explained, the files will be downloaded to the current directory in the command prompt at the time of execution. Thanks very much, again. If I may follow up with another question - could you also only download selected edf files with the nsrr gem? I tried some variations of the nsrr download command but got nowhere. Thanks!

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · almost 9 years ago

Hi Stephany, currently there is no method for single file downloads, outside of directly downloading them through the browser. However we have had this requested in the past, and may be adding this in a v0.3.0 of the nsrr gem. I'll send you a message when the functionality has been added. Thanks for the feedback!

edit: v0.3.0 is now available!

4 posts
Was this reply useful? Learn more...
 
[-]
hardikmadhu +0 points · over 8 years ago

Hi,

My access for shhs database has been approved. But when i try to download with nsrr command as per the posts above, and copy paste my token, it always gives me this error. "UNAUTHORIZED Public Access Only".

Is there anything that I am missing ?

Thanks in Advance, Hardik

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · over 8 years ago

Dear Hardik,

Sorry for the inconvenience, I noticed an issue with your access token and have reset the token. Can you give the NSRR gem another try, it should now correctly download the files from SHHS. You token has also changed, so you'll need to visit https://sleepdata.org/token to get the new one.

Thanks!

Remo

4 posts
Was this reply useful? Learn more...
 
[-]
hardikmadhu +0 points · over 8 years ago

Now it works. Many thanks Remo.

Hardik

3 posts
Was this reply useful? Learn more...
 
[-]
aekhadj +0 points · over 6 years ago

I did not understand how to download the database shhs1?

3 posts
Was this reply useful? Learn more...
 
[-]
aekhadj +0 points · over 6 years ago

Dear remomueller I did not understand how to download the database shhs1? Help me please

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · over 6 years ago

Here's an overview of the steps users take to request data on the NSRR: https://sleepdata.org/showcase/where-to-start

442 posts
bio
Was this reply useful? Learn more...
 
[-]
mrueschman +0 points · over 6 years ago

Aek,

You must have an approved Data Access and Use Agreement before you can download data from the Sleep Heart Health Study (SHHS). We have received your four (4) DAUA requests and will notify you of our decision next week.

3 posts
Was this reply useful? Learn more...
 
[-]
nadisadr +0 points · over 6 years ago

Do I need to request access to download SHHS2 as well? It's mentioned in the PhysioNet webpage that "Following a 2003 agreement between SHHS and PhysioNet to make 1000 polysomnograms from the SHHS-2 examination available freely via PhysioBank, in 2014 the SHHS investigators launched the National Sleep Research Resource (NSRR), which offers free access to much more of the SHHS PSG Database". There's only a sample record available to download. I just need 100-200 records or even less than 100 from the SHHS-2 would work for my research. I was wondering where I can download them for free and if I still need to request access and provide an IRB review for that.

52 posts
bio
Was this reply useful? Learn more...
 
[-]
remomueller +0 points · over 6 years ago

Hello Nadi, the SHHS dataset on the NSRR contains both the SHHS1 and SHHS2 visits. I've linked the polysomnogram folders that are available, but there are also CSV datasets that cover other data collected. Requesting access to the SHHS dataset will give you access to both datasets for free. Let me know if you have any further questions!

3 posts
Was this reply useful? Learn more...
 
[-]
nadisadr +0 points · over 6 years ago

So it seems that I had to put a request anyway, which I just did. Hopefully, I can get the access soon. Thanks.

442 posts
bio
Was this reply useful? Learn more...
 
[-]
mrueschman +1 point · over 6 years ago

Note that IRB review is not a firm requirement (though it's preferred) for SHHS data access. There is a second option to select for NSRR Internal Review when completing the Data Access and Use Agreement process. I look forward to reviewing your submission!

3 posts
Was this reply useful? Learn more...
 
[-]
nadisadr +0 points · over 6 years ago

Thanks for the information. I just submitted my request and look forward to hearing from your review. :)