Skip to main content

How to mirror/sync a remote ftp folder with a local folder

lftp is one of the best command line utilities currently available for working with ftp servers, One of its useful features is mirroring which is suitable for backup purposes and very useful for slow connections. In this article i'm going to show you how to use this utility to backup and mirror your files a ftp server.

Simple Usage

The following script mirros your local folder to a remote ftp server, it only updates the changes you've made since the last execution so only the first time requires lots of bandwidth :

#!/bin/bash
HOST='mysite.com'
USER='myuser'
PASS='myuser'
TARGETFOLDER='/new'
SOURCEFOLDER='/home/myuser/backups'

lftp -f "
open $HOST
user $USER $PASS
lcd $SOURCEFOLDER
mirror --reverse --delete --verbose $SOURCEFOLDER $TARGETFOLDER
bye
"
You can save it as upload.sh and execute it using the following command :

bash upload.sh
Advanced Usage

lftp offers many parameters, here i'll explain some of the most useful ones :

--reverse : Indicates that you want to update your remote folder. if you want to update your local folder then you should remove this option and also swap values of TARGETFOLDER and SOURCEFOLDER variables
--delete : It tells lftp that you want to remove all the files/folders that no longer exist on source folder
--use-cache : If you're about to sync very large number of files, this option can speed up things a lot. It does not have any effect the first time but in the next sessions it uses local cache instead of scanning the remote server
--exclude : Sometimes there are folder/files on both source and target folders that have nothing to do with the mirroring, using this option we can tell lftp to ignore this sort of files
Tips

FTP servers' version and configuration is not always the same , so the above scripts might not always work,

The usual issue is Passive Mode, and occurs when the client (you) is behind proxy or firewall.

Comments

  1. Prefer using rsync or wget for this, to transfer data securely(even from a remote ftp folder) and to make sure that minimum data transfers occur on slower connections.
    Since rsync uses delta to update any files that may already be present in the folder you are about to sync this is always a preferable method.
    Cheers!
    Abhi

    ReplyDelete
  2. Thanks Avi for your suggestion. You can post your preferred code for the same as well.

    ReplyDelete

Post a Comment

Popular posts from this blog

Shell Script to check tomcat status and restart

The below script checks the status of a particular tomcat status and restarts it if the tomcat does not respond....TOMCAT_HOME=/usr/local/tomcat-folder/

is_Running ()
{

        wget -O - http://yourserver.com/ >& /dev/null
 if( test $? -eq 0 ) then
  return 0
 else
  return 1
 fi
}


kill_Hanged_Processes ()
{
 echo "killing hanged processes......"
 javaProcs=`ps -efl| grep -v grep | grep java`
 if(test ! -z "$javaProcs") then
  echo "nonzero"
  processId=`echo $javaProcs | awk '{ print $2} '`
  echo "$processId"
  kill -9 $processId
 fi
}
stop_Tomcat ()
{
 echo "shutting down......"
 $TOMCAT_HOME/bin/shutdown.sh
}

start_Tomcat ()
{
 echo "starting......"
 $TOMCAT_HOME/bin/startup.sh
}

restart ()
{
 stop_Tomcat
 sleep 10
 kill_Hanged_Processes
 start_Tomcat
 sleep 60
}

send_Mail ()
{
#!/bin/bash
# script to send simple email
# email subject
SUBJECT="Telegraphindia.com went down"
# Email To ?
EMAIL="mailid1@gmail.com,mailid2@gmail.com"
# Em…

Getting access to menu from portlet in liferay

The below code describes the way of accessing menu items from the portlet. Here i have called it from jsp. The best thing about this is you can use the beautiful permission management of menu items of liferay as well without bothering about that. Writing services for fetching menu names from the layout tables will need more codes for permission managements......

Hope this helps.........


<%--
/**
* Copyright (c) 2000-2010 Liferay, Inc. All rights reserved.
*
* This library is free software; you can redistribute it and/or modify it under
* the terms of the GNU Lesser General Public License as published by the Free
* Software Foundation; either version 2.1 of the License, or (at your option)
* any later version.
*
* This library is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
* details.
*/
--%>
<%@ page import="com.li…

Virus f Opyum Team

Symptoms :: 
Several processes running named f or i. 
Network choked
Apache Dead
Unknown entry  * * * * * root f Opyum Team in crontab

Investigate ::
Try to kill the f process and even root will be unable to kill it. Check file named f and i in /etc or /bin or /usr/sbin directory. Try to delete, the file f  will not delete.  Some unknown .jpg files will be there in the same folder. 

Solve :: The root is unable to delete the file f because of the immutable bit in f
To remove it use ses or chattr
Follow the below steps to remove it completely.....



remove the unknown .jpg files in system folders

#top
(kill process f) option k

#ses -i /bin/f
#rm /bin/f
#ses -i /etc/crontab
#vi /etc/crontab (and delete last line)
#reboot


Precautions ::  In my case the server was hacked bu Brute-force attack
So set a good password with a combination of upper,lower case letters and special characters.

Post your feedback or any issues you may face removing the virus.  Very less number of good threads are there explaining …