I did the migration as described above. But when I login into HPC using hpc-login2.usc.edu, the /project directory is not there (whereas it is there when I login using hpc-transfer.usc.edu).
Hi @hrayrhar, this is intentional. We want to make sure that people get good transfer rates so access between the old /home/rcf-proj and new /project was only set up on hpc-transfer.usc.edu
Two ways to find the path to the project directory you belong to:
- Using the
myquotacommand on the command line. - On the User Portal homepage (hpcaccount.usc.edu):
I run into an rsync error code 23 saying that there are some files failed to be migrated, because of permission denied. How can I change permissions of these files in the rcf-proj?
Do I need to migrate my data under /home/rcf-proj2?
@kerenxu That sounds like you don’t have read permissions for those files. Can you check who owns those files/directories and what the permissions are? You can use ls -ld /path/to/file, for example. The owner can either transfer those files or change the permissions to allow you to transfer them, using the chmod or setfacl commands. You would need read permissions for files and read/execute permissions for directories.
1 Like
I migrated another user’s folder from /rcf-proj to the new /project/<PI_name> directory but it now has my user as the owner. How can I change the directory’s ownership back to the other user? I tried using chown but got an “Operation not permitted” even though I was the current owner.
I got the following error (please see attached screenshot)
How can I find the files not transferred? I migrated my entire rcf-proj/xxx folder so don’t know an efficient way of finding the missing/non-transferred files.
Thank you
1 Like
It’s possible some files were not transferred due to a permissions issue. You can check by re-running rsync and directing the output to a log file. Then you can search through the log to see which files weren’t transferred and check permissions.
You can do something like this:
rsync -rltvP /home/rcf-proj/xxx/yyy/ /project/aaa/bbb/ &> rsync_log
That should create a file named rsync_log which you can look through and find which files caused the issue.
1 Like
thank you, Cesar! i will do this
1 Like
I have an HSDA account, and I tried to transfer the files from our rcf-proj folder and it said permission denied. Am I supposed to transfer these files? I also have an rcf-40 account, will those files be okay
HSDA data should stay on the login-pd system. Any data saved in your home directory, /home/rcf-XX/username will need to be copied over to your new home directory, /home1/username, assuming it is not HSDA data.
I have been able to create a new directory under the new project directory, but for some reason when running the rsync command I can’t seem to access anything in the /rcf-proj folder. It just keeps saying no such file or directory. I know I have the right directory, because I can access and see in the rcf-proj while on the head node.
Can anyone point me in the right direction?
Thanks!
@rsschell Could you share the command you are using? Does it transfer some files? This could be a permissions issues. Are you the owner of the files?
Yes I am the owner. I am a ‘user’ in the /home/rcf-proj/ime folder but the owner of /home/rcf-proj/ime/rsschell
This is the command:
rsync -rltvP /home/rcf-proj/ime/rsschell/ /project/ehrenrei_315/rsschell/
This is the output/error:
sending incremental file list
rsync: change_dir “/home/rcf-proj/ime/rsschell” failed: No such file or directory (2)
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
But also even ls or trying to cd to the /home/rcf-proj/ime/rsschell/ folder on the transfer node produces this error:
cannot access /home/rcf-proj/ime/rsschell/: No such file or directory
Meanwhile copy paste the same code on login2 I can look and cd to that folder.
We have extended the deadline to Sunday, October 11 before midnight . If you haven’t already done so, please complete your data migration this week/this weekend.
Okay, that all looks correct. Which transfer node are you on? This should work only from hpc-transfer.usc.edu (not hpc-transfer1/2).