This paper aims at developing a flexible video analysis system that can be used in wide range of video surveillance applications as well as to detect the human being. The developed system is called here as Smart Video Analysis System. This SVAS is able to detect and track interested objects. It can also detect people and recognize their activities in an application environment, such as in a room, supermarket, car, or security checkpoint. Designing a real-time video analysis system is a complex task, as many factors including processing speed, system cost, accuracy, and robustness, need to be carefully balanced. This research has focused these factors at two levels, algorithm level and software level. Background elimination algorithm is proposed in this paper to enhance the performance of Smart Camera systems in changing background and varying lighting condition environment. Finally, the software implementation of the Smart Camera Analysis Systems is applied to detect motion and then to trigger alarm for the security purposes. The system will trigger alarm once the motion is detected and when it exceeds the desire threshold value.