Lecture notes Linux operating system

linux programming lecture notes
NathanBenett Profile Pic
NathanBenett,Germany,Researcher
Published Date:11-07-2017
Your Website URL(Optional)
Comment
Basic Introduction to UNIX/linux Claude Cantin (claude.cantinnrc.ca) http://www.nrc.ca/imsb/rcsg Research Computing Support Group Information Management Services Branch National Research Council July 9, 2017Chapter 1 Introduction ThiscourseisintendedforpeoplenotfamiliarwiththeUNIX/linuxoperating system, butfamiliarwithothercomputersystemssuchasMSWindows,DOS orVMS.Itismeantasanintroductionforbeginnerstohelpthemunderstand concepts behind the UNIX/linux operating system. Intermediate users may find the course useful as a refresher. Up to 2003, most of the command examples used throughout the text 2 were performed using a Silicon Graphics O , running IRIX 6.5. But since September 2003, the hands-on portion of the course is done using the linux (SuSE8.2,then9.0)operatingsystem, whichmeansmostcommandsarenow done with linux. SGI systems running IRIX, Sun systems running Solaris, Hewlett-Packards running HP/UX, IBM RS/6000s running AIX and most PCs (and other architectures) running linux use most of the commands de- scribed in this manual. They use the same basic commands, although some of the options used may vary slightly between the different architectures. In specific cases, the book uses commands based on linux. The distribu- tion used was SuSE version 7.3 and newer. NOTES: • This book refers to various UNIX derivatives running on “worksta- tions”. The author’s definition of “workstation” includes systems such as the Sun Microsystems SPARCstation family, the Silicon Graphics Personal 2 IRIS,Indigo,Indigo ,PowerSeries, Challenge, PowerChallenge, Onyx, 2 Power Onyx, Indy, O , Octane, Origin and Altix families, the IBM 34 CHAPTER 1. INTRODUCTION RS/6000 series, the HP 9000 model 700 and 800 families, the Com- paq AXP families (systems running True UNIX), and 500+ MHz PCs running one of the linux distributions. • AlthoughmostsectionsrefertoUNIXingeneral,somerefertoaspecific architecture. Others may refer to NRC-specific topics. Those sections are generally clearly indicated. • This book also refers to various linux distributions, notably SuSE 7.x and 8.x, and Red Hat 7.x and 8.x. 1.1 History The UNIX story begins in 1969, when AT&T Bell Labs dropped out of a joint project between themselves, General Electric, and the Massachusetts Institute of Technology. Multics, the project in question, was an experi- mental operating system on the GE 645 (thanks to Tom Van Vleck from http://www.multicians.org for correcting details from pre-2002 versions of the course notes). Ken Thompson and Dennis Ritchie, both from Bell Labs, had been ex- posed to the Multics project. They ported a game, called ‘Space Travel’, from the GE 645 running Multics, onto a PDP-7. To help them do the port- ing, Thompson wrote a“simple file system and some utilities forthePDP-7” 8, p. 3. This was the birth of UNIX, in 1970 (“UNIX” was meant to be a pun on “Multics”). Late in 1970, a PDP-11/20 was purchased, and UNIX became an official Bell Labs project. The first UNIX edition “was documented in a manual authored by Thompson and Ritchie dated November 1971”1, p. 2. Most of the ideas found in today’s UNIX systems were incorporated in this edition. The second edition, 1972, incorporated the piping concept. In 1973, UNIX was re-written in C, by Thompson and Ritchie. Note that Ritchie developed the C language (derived from the B language by Thompson) at approximately the same time. UNIXwasfirstdistributedinMay1975foranominalfee. ThiswasUNIX Version 6. In 1979, a more portable version of UNIX (Version 7) was released for general use; from here, three major UNIX versions emerged: BSD (Berkeley System Distribution), XENIX, and AT&T’s System V.1.1. HISTORY 5 1.1.1 BSD: Berkeley System Distribution The University ofCaliforniaatBerkeley acquired Version 4ofUNIX in1974. In 1975, during a sabbatical leave from Bell Labs, Ken Thompson went to Berkeley and helped install UNIX Version 6 on a PDP-11/70. “The same year, twograduatestudents alsoarrivedatBerkeley: BillJoyandChuck Ha- ley” 8, p. 5 : they were to play an important role in the BSD developments. Joy wrote the vi editor, and the C shell. Bill Joy is also one of the co-founders of Sun Microsystems. In 1978, 2BSD was released. In 1979, a combination of improved 2BSD and UNIX Version 7 was re- leased as 3BSD. DARPA (the Defence Advanced Research Project Agency) funded the development of 4BSD and 4.1BSD. The Fast File System was included in 4.2BSD which was released in 1983. Finally, 4.3BSD was released in 1987. Berkeley’s development ofitsBSDUNIXendedwiththereleaseofitslast major version, 4.4BSD, in 1992. Then, late in 1994, it released 4.4BSDlite. Many UNIX variants are based on thatlast version, written torun better onPC’sbasedontheIntel’486andPentiumCPUs,namelyFreeBSD,NetBSD, BSD/OSandOpenBSD.Newversionsoftheseversionsbecameavailablein1995. 1.1.2 XENIX XENIX is directly based on UNIX Version 7. Its first version, 2.3, was releasedin1980byMicrosoftforuseonmicrocomputers(IBMPCsorclones). XENIX 3.0, released in 1983, included new features from 4.1BSD and from AT&T’s System III. Finally, thelatestversion, XENIX 5.0was released in1985,andconforms to the System V Interface Definition. Xenix is not used anymore. 1.1.3 Linux linux is a version of UNIX (some would say a “UNIX clone”), based on the original kernel work by Linus Torvalds in Helsinki, Finland. It runs mostly on the x86 and Pentium family of micro-processors, but has been ported to many other architectures.6 CHAPTER 1. INTRODUCTION The original kernel work by Linus Torvalds was done in the early 1990s. Full linux distributions first appeared circa 1993. They were (still are) based on Torvalds’ kernel, but consisted mostly of software/applications openly available for anyone wishing to use it. Different distributions of linux include Red Hat, Slackware, Caldera, Debian, SuSE, Mandrake and many others. Although linux may be downloaded at no charge (for free), it has a copyright. It follows the GNU General Pulic License (GPL), in that it is freely available, but if sold, a copy of the source code must be provided. Details on GNU’s GPL is available at http://www.gnu.org/copyleft/gpl.html. Today(2003),linuxisthefastestgrowingsegmentofanyUNIXoperating system variant. 1.1.4 System V (formerly known as “AT&T” System V) Because of a “1956 consent decree, AT&T was limited in the businesses that they could engage in and the patents they could licence...” 8, p. 4. This consent decree delayed the introduction of AT&T’s first commercial release by a few years. This first official release finally occurred in 1982, and was called System III. This version did not include “important Berkeley innovations, such as the C shell and screen editing capabilities” 8, p. 6. The next version, System V, followed in 1983. This version did include utilities such as vi and curses from the BSD. System V Release 2 was released in 1984, and System V Release 3 in 1987. Finally, System VRelease 4 was officially announced on Nov 1, 1989,and was released in 1990. This version combines System V Release 3, 4.3BSD, XENIX, SunOS and new features in one operating system. Figure 1.1 offers a graphical view of the evolution of the UNIX operating system. 1.2 History (continued) In 1988, two major competing UNIX groups emerged: the OSF (Open Soft- ware Foundation) and UI (UNIX International). OSF was lead by IBM,1.2. HISTORY (CONTINUED) 7 Figure 1.1: History of UNIX 1970 UNIX 1974 Version 4 BSD Version 5 2BSD 1979 Version 7 2.9BSD 1981 System III Xenix 2.3 3BSD 1983 System V Xenix 3.0 4BSD 1985 System VR2 Xenix 5.0 1987 System VR3 4.3BSD 1990 System VR4 Linux 4.4BSD 1995 (Solaris 2.4) SuSE 4.4BSDlite (IRIX 5.3 ) RedHat BSD/OS Debian FreeBSD Slackware NetBSD 2000 (Solaris 8) ... OpenBSD (IRIX 6.5)8 CHAPTER 1. INTRODUCTION Digital, and HP, whereas UI was lead by AT&T and Sun Microsystems. TheOSFwasfirstformedtocounterAT&T’salliancewithSunMicrosys- tems to license and enhance the UNIX Operating system. OSF wanted to write their own version of UNIX, based on IBM’s AIX. UNIX International was formed shortly aftertopromote AT&T’s System V UNIX system. The “special” alliance between AT&T and Sun Microsys- tems was reduced to partnerships between all UI members. InJanuary1990,UIanditsmemberswereapplaudingthereleaseofUNIX System V release 4 (SVR4, or V.4), which combines SVR3, BSD, XENIX, and SunOS. Early in 1993, Sun was shipping Solaris 2.1, completely based on SVR4. Silicon Graphics starting shipping IRIX 5.0 in the spring or 1993 and IRIX 5.2 for all their systems in early 1994. IRIX 6.0, a 64bit version, was shipping for the high-end systems in in the fall of 1994. IRIX 6.2 is SGI’s “all-platform” newest release, released in the Spring of 1996. In January 1993, Novell purchased USL thus taking control of UNIX. In September 1993, Novell re-organised and formed the UNIX Systems Group (USG), which took control of UNIX. In late 1995, Novell sold the control of UNIX (including UnixWare, Sys- temVRelease4.2code)totheSantaCruzOperation(SCO).In2001,Caldera Systems(alinuxvendor)purchasedSCO’sServerSoftwareDivisionaswellas SCO’s Professional Services Division and became Caldera International. What was left of SCO became Tarantella, now mostly a “professional ser- vices” company. In 2002, Caldera International, Inc. changed its name to The SCO Group. Andin2003,TheSCOGroupinitiatedaseriesoflawsuitsagainstseveral UNIX/linuxvendors, claimingUNIXcopyrightedcodehadmadeitswayinto linux. Four years later, in 2007, it was ruled that Novell (now also owner of the SUSE linux distribution) owned the UNIX copyright; that decision was overrulled in 2009. In 2010, the courts once again confirmed Novell was the rightful owner of the UNIX trademark, and in 2011 an Appeals Court agreed. By the time that was done, SCO had gone bankrupt and re-emerged (in 2008) as a privately traded company. Most workstation vendors’ current 64 bit Operating Systems: SGI with IRIX 6.5, Compaq (formerly Digital) with Tru64 UNIX (Compaq was ac-1.3. THE UNIX OPERATING SYSTEM 9 quired by HP in 2002), HP with HP/UX 11, Sun with Solaris 9. 1.3 The UNIX Operating System The UNIX system is mainly composed of three different parts: the kernel, the file system, and the shell. The kernel is that part of the system which manages the re- sources of whatever computer system it lives on, to keep track of the disks, tapes, printers, terminals, communication lines and any other devices. The file system is the organising structure for data. The file systemisperhapsthemostimportantpartoftheUNIXoperating system. The file system goes beyond being a simple repository for data, and provides the means of organizing the layout of the data storage in complex ways. The shell is the command interpreter. Although the shell is just a utility program, and is not properly a part of the system, it is the partthattheuser sees. The shell listens toyour terminal and translates yourrequests intoactions onthepartofthekernel and the many utility programs. 7, p. 3 file systems arediscussed in chapters 2and3ofthis manual, andshells are discussed in chapter 4. One can imagine the UNIX system as a series of three concentric circles, with the inner circle representing the kernel, the second circle representing the programmingshell, and the last one representing application programs. Figure 1.2 illustrates that concept. The shell communicates to the kernel, and vice versa. The application programs can communicate directly with both the shell and the kernel. 1.3.1 Multi-tasking, Time Sharing UNIX is a multi-tasking operating system, which means that a number of programs can run at the same time. Those programs (calledprocesses) can communicate with each other.1.3. THE UNIX OPERATING SYSTEM 11 For example, a C program could be compiling as mail is being read or a file is being edited. Processes that “wake-up” occasionally, and/or regularly, are called dae- mons. Daemons are used to synchronise disks, send and receive mail, print documents, and so on. 1.3.2 Multi-user UNIX is also multi-user: two, three, or more users are able to use the same processor to execute their programs. 1.3.3 Network Capabilities Today’s UNIX workstations come with TCP/IP and ethernet connections. The same is true for PCs. At NRC the ethernet network connects dozens of Suns,SiliconGraphics,VAXes,IBMRS/6000s,HPs,andmostPCstogether. From any of those nodes, it is simple to logon to a remote machine, send mail to a user on those machines, or transfer files to or from those nodes. More on communications in a later chapter. 1.3.4 Portability Traditionally, most operating systems were written in Assembler, for a spe- cific architecture. It was therefore VERY painful – if at all possible – to ‘port’ the operating system to other architectures. UNIX, on the other hand, is mostly written in the C language. This aloneallowsUNIXtobeportabletomanyarchitectures. Today,UNIX/linux runs on more architectures than any other operating system in the world. Examples of such architectures/processors are the Motorola 680X0-based workstations, the 80X86 machines, the RISC based architectures (SPARC, MIPS, 88000), VAXes, IBM mainframes, Amdahl, Cray, and many more. And this does not include all the different architectures linux runs on. Table 1.1 lists different UNIX versions vendors use. 1.3.5 Flexibility UNIX is also a very flexible operating system, both for system administra- tors and users. Program names can be changed. aliases can be defined.12 CHAPTER 1. INTRODUCTION UNIX variant Vendor SunOS Sun Microsystems Solaris Sun Microsystems IRIX Silicon Graphics Inc. AIX IBM HP-UX Hewlett Packard OSF/1 HP (Compaq/Digital) Digital UNIX HP (Digital) A/UX Apple Computers SCO Santa Cruz Operation Destiny AT&T XENIX Microsoft UTS Amdahl UniCos Cray UXP Fujitsu Linux Public domain FreeBSD Public domain NetBSD Public domain Table 1.1: UNIX variants on the market. Arguments to programs can also be changed. New programs can be built, and put in the user’s own bin directory, thus allowing further customisation of the system. 1.3.6 Software Available Thousandsofapplicationpackages12,p.2areavailablefortheUNIX/linux system. In addition to the commercial packages, many programs are written and made available in the public domain. Examples of such packages are the X Window system, written at the Massachusetts Institute of Technology, the T X system produced at Stanford, and many other utilities/applications E written by individuals and organisations, forthebenefit ofthe“Open Source Community”. For many, it is their way of thanking the “Internet Commu- nity” for the vast amount of resources available.1.4. UNIX PHILOSOPHY 13 Linuxisaperfectexamplesoftheincredibleamountofsoftwareavailable, at no cost to individuals. In fact, most open source, freeware, or public domain packages used in the scientific world are developed and maintained on a linux platform. The same can be said of any open source, freeware, or public domain packages developed on any UNIX variant: linux is the platform of choice to develop/write software. 1.3.7 Virtual Memory TheUNIXoperatingsystemhasvirtualmemory,orswapspace,whichmeans it can run programs bigger than the amount of RAM the computer actually has The amount of virtual memory is decided upon by the system adminis- trator. 1.3.8 Case Sensitivity The most common mistake for beginners involves the use of mixed case in UNIX commands: UNIX is case sensitive, i.e., “a” is different from “A”. 1.4 UNIX Philosophy UNIX’s philosophy is the same as the C language’s: it assumes users know what they are doing 1.5 Exercises 1. What is the operating system that covers the widest range of architectures? Why? 2. What are some of the advantages of running UNIX? 3. What is UNIX’s main philosophy? 4. Name some of the reasons people would install and use linux on their desktops or servers.Chapter 2 File System The UNIX File System manages and controls access to files and directories. It keeps track of opened and closed files, and manages files on the hard disk. 2.1 File and Directory Names 2.1.1 Length The maximum length of directory and file names varies from system to sys- tem. Some versions (such as HP’s HP/UX) may restrict the names to 14 characters, whereas most allow much longer file and path names. For example, Sun and SGI workstations allows file names up to 255 char- acters long, and path names to be as long as 1023 characters (a path name is the full length of the file name, beginning at the top – root – of the file system). 2.1.2 Conventions Naming files is very simple: anyASCII character canbeused. It is, however, recommended that no metacharacters (∗,,, , , ?, ,\, , , ,, &) be used. It is also recommended to simply use letters, digits, underscore ( ), hyphen (-) and the dot (.). Byconvention, filesendingwith.careClanguagefiles. Filenamesending with .p, .f, .s are respectively PASCAL, FORTRAN, and assembler program files. Header (include) files usually end with .h. 1516 CHAPTER 2. FILE SYSTEM File names beginning with a . (dot) are hidden files: they do not appear when files are listed. Those files are typically configuration files. Usually, their name consists of the package they represent followed by the rc (run command) string. .mailrc and .newsrc are two examples. Any file name may contain several dots and/or underscores. These are NOT separators. 2.2 Structure of Directories, Files The filing structure of the UNIX operating system is hierarchical. It is a tree structured system, completely open (assuming necessary permissions) to every user on the system, with everything emerging from / (root) at the top. Figure 2.1 illustrates this concept. Every directory has a parent, and—possibly—one or more children. Those children can in turn be parents. A directory is a special type of file. A file and a directory of the same name within the same directory is therefore impossible. NOTE that a file is a linear sequence of characters, including line feeds (\n); there is no specific file structure. EverythinginUNIXisconsideredafile;adirectoryisaspecialkindoffile, andsoisthekeyboard(/dev/kbd)andthesystem’sconsole(/dev/console). 2.3 Permissions/File Access Modes A permission defines the ownership of a file and the access of all users to that file (or directory). Each file and directory has permissions associated with it. Those permis- sions are made up of three groups of three characters: rwx rwx rwx (read, write and execute). The first group represents the owner permissions on the file, the second group represents the group permissions on the file, and the third group represents the world permissions on the file. The owner is the user owning the file. Each user has an internal user number (UID), and an internal group number (GID), set by the system administrator. Everyone with the same2.3. PERMISSIONS/FILE ACCESS MODES 17 (root) etc tmp usr dev var passwd hosts tmpfile lib people bin bsd adm mail spool tmp libc.a libm.a class1 cantin podaima vi talk bin prog .cshrc Figure 2.1: Structure of File System group number is said to be in the same group. Group permissions apply to everyone else with the same group number. world is everyone else. For example, the file phone.numbers may have permissions set as rwxrwxr. This would mean that the owner of the file can read the file, write to it, and execute it. People in his/her group have the same privileges. Everyone else can only read the file; they cannot change, or execute it. The permissions of a file/directory can be changed with the chmod com- mand.18 CHAPTER 2. FILE SYSTEM 2.3.1 chmod: Change Mode (Permissions) The chmod command allows a user to change the permissions of a file/directory. To use chmod, the user must be the owner of the file. The syntax of the command is: chmod -R mode filename(s) -R is an option. -R (Recursively) will cause all files and directories within (underneath) the file/directory whose permissions are being changed to take those permissions. mode maybe specified asthreeoctalvalues (oneforeach ofthe threesets ofpermissions): ifanyofthepermissionbitsr,w,orxisset,thecorresponding permission is enabled: give it a 1. If not, give it a 0. Then, for each of the three permission groups, interpret the three binary numbers as an octal number. Another way to explain it is to give different weighing factors to the different permissions: 4 to “r”, 2 to “w”, and 1 to “x”. If the permission is set, add the weighing factor. Otherwise do not add anything to the group value. Using the above example of the file phone.numbers, which had the rwxrwxrpermissions,itwouldtranslateto111 111 100,or774,or4+2+1 4+2+1 4+0+0: rwx rwx r 111 111 100 7 7 4 If write permission for world (sometimes called others) is added, mode is changed to 776 (or 111 111 110, or 4+2+1 4+2+1 4+2+0). mode may also be specified symbolically as +r which would add read permissions to everyone, or as -w which would take away write permissions from everyone (all groups). filename(s)maybeoneormorefilenames, and/oroneormoredirectories, separated by blank spaces. For example, to change the permissions on file phone.numbers, from rwxrwxr to rwxrwxrw-, the command2.3. PERMISSIONS/FILE ACCESS MODES 19 chmod 776 phone.numbers or chmod +w phone.numbers or (because we are changing it only in the others group) chmod go+w phone.numbers which means “group others add write”, could be used. The general syntax, using the conventional method is chmod ugo+-rwx filename(s) Where u : permission for user/owner. g : permission for group. o : permission for others. r : read permission. w : write permission. x : execute permission. Any combination of ugo, rwx may be used. If none is used, then all three are assumed. One exception: if chmod - is used, the result depends on the value of umask (umask sets the default permissions on files; usually 022 at NRC – see man page for details). It is also acceptable to put a combination of “ugo+-rwx”, as long as they are separated by a comma (,), as in chmod ug+x,o-x filename . Typing the command chmod help (linux) or chmod (generic UNIX) by itself will display the usage of the command. Refer to the UNIX manual of your machine for more options on the command.20 CHAPTER 2. FILE SYSTEM 2.4 Exercises 1. My friend just created a file in his/her directory. Can I go and change permissions on it? Why, or why not ? 2. I just created a new file in my directory, but I would like it to have rwx permissions for me, and r permissions for everyone else. What command can I use to change those permissions? 3. I have a file with permissions rwxrwxr-x. I want it to have rwxr-xr-x permissions. What commands (I want two) can be used to change the permissions on it? 4. Take the same file. I want to remove all execute permissions for every- one, except myself. Again, which command can be used to achieve my goal?Chapter 3 Tour of the File system 3.1 /: Root / by itself is the root or the top, the beginning of the file system. “super user” is usually the only user allowed to write to that directory. Each direc- tory contains files (and directories) related to one group of subjects. Some of those directories are discussed below. For more information on any of them, please see 3, pp. 17-48. 3.2 /bin and /usr/bin: Commands The /bin and /usr/bin directories contain public commands. Those com- mands may be in binary, or in shell format. Binary format commands are those written in C, C++, FORTRAN, or any other compiler language. The shell format commands are interpreted by the C or Bourne shells, whichever executes them (more on shells later). When UNIX was first written, /bin and /usr/bin physically resided on twodifferentdisks: /binbeingonasmallerfaster(moreexpensive) disk, and /usr/bin on a bigger slower disk. Now, /bin is a symbolic link to /usr/bin: they are essentially the same directory. 2122 CHAPTER 3. TOUR OF THE FILE SYSTEM 3.3 /dev: Devices In UNIX, every output device is referred to as a file. For writing to the console (the monitor), the file to write to is /dev/console. To write to tape, the file is /dev/nrst0 (on a Sun SPARCstation 2). Even the keyboard writes to a file (/dev/kbd) which is read by the operating system. 3.4 /etc and /usr/etc: Management The /etc and /usr/etc directories are the “management” directories. /etc and /usr/etc contain programs used by the system administrator to admin- ister and configure the system to the needs of the users. One of the most important files of the /etc directory is the passwd (pro- nounced “password”) file. The passwd file contains basic information about each user’s account, including the logon and, in some cases, the password (encrypted) of every user (in the other cases, the encrypted passwords are in a file called /etc/shadow which only the super user has access). Again, the commands are in two different directories simply for historical reasons. 3.5 Home Directories The HOME directory is the directory in which the user lands when logging into the system. Different vendors use different conventions for their HOME directory. Vendor HOME directory HP /users/login IBM /u/login Linux /home/login Silicon Graphics /usr/people/login SUN (SunOS) /home/machine name/login SUN (Solaris) /users/login Table 3.1: HOME directories3.6. /LIB AND /USR/LIB: LIBRARIES 23 login refers to the user’s account name whereas machine name refers to the name of the system where the account resides. 3.6 /lib and /usr/lib: Libraries “Theseareaswereoriginallyintendedforlibrariesofcompiledsubroutines, to besearched duringthelinkingphase followingthecompilationofaprogram” 3, p. 27. Those areas now also contain programs not directly available to users but used by other programs, such as compilers. They also contain databases (fonts, conversion units, etc.) used by programs. IRIX5.2(fromSGI)has5filesin/lib. /usr/libhasafewthousand. On other systems, like Suns, /lib points to /usr/lib. Hence, both directories share the same information. 3.7 /tmp: Temporary Directories /tmp is a necessary “temporary” directory. It is available to all users to read from and write into. But, along with /var/tmp and /usr/tmp, it is mostly used by compilers and by UNIX as temporary storage for intermediate files. 3.8 /var: Directories The philosophy behind the /var directory structure is that it contains direc- tories that vary in size and that tend to grow if not maintained properly. 3.8.1 /var/adm: Administration Administration information is kept in this directory. Logins—who logged on when and for how long—are also kept here. Messages output to the console, system crashes and accounting information for each user are kept in specific files that grow continuously. The system administrator can monitor which commands are most often used by referring to database files from this directory. He/she may find out how much CPU time a specific account uses, even the commands used by specific users (if the necessary configurations have been made).

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.