Hi Gang,
I asked another question around one month ago and even though I could not
come up with a answer I decided to come up here again.(Incidentally I
tried it also in linuxquestions.org).
What would be the equivalent to the zsh script given below in bash
$echo ${${(z) $(whereis libcrypto)}[2]}
if the output of $(whereis libcrypto) is
libcrypto: libcrypto.so libcryto.a
it returns libcrypto.so only.
Now I could come up with
$robin=($(whereis libcrypto)); echo ${robin[1]}
But can it be done in one go using some construct?
I am not a shell guru so I wonder if it can be done!
Wed, 31 Dec 2003 04:05:55 -0800
Dave Bechtel (kingneutron
from yahoo.com)
Answered By Heather Stern, Ashwin
Dear God, yahoo really b0rked that one. All because I wanted to
underline .net... Sorry 'bout that.
--I've been vaguely interested in clustering for a while, but really
don't know what I could do with it. I have 3 machines:
P166 Dell laptop (Intel) 128MB
P233 Squid / DSL server (Intel) 256 MB
900 MHz Duron desktop (AMD) 512 MB
--I can't recompile a kernel intended for the slow Intel machines on
the fast AMD, it just doesn't work. However, I could go out and buy
some cheap machines at a computer show for ~$30-$40 apiece, or get
some loaners from a friend.
--My question is, what are clusters good for besides graphics /
animation / video editing / number crunching (none of which I'm
interested in)? Can I gzip/ bzip2 compress a 4-8GB tar file on a
distributed cluster and save time? Can I recompile a 2.4 kernel that
way so it doesn't take an hour and a half? (Yes, even using "make -s
-j 3".)
--I've heard of OpenMosix, but haven't looked into it very far. Just
a note, my network is 100MBit Ethernet. Any info the LG.net folks
could provide would be welcome, TIA.
[Heather]
We'll need permission to publish your message and responses related to
the thread for the world wide webzine Linux Gazette, it you want a
decent chance at an answer.
This is clear enough to post as a Wanted if none of the Answer Gang want
to take a pop at it, but we're past deadline for the current issue; it'd
end up in February.
And of course, he did grant that, and here you have it.
But it's worth noting to folks who copyright their usenet posts,
we need your intent to be public or we can't publish it :) And
if we can't publish it, we often won't answer it either.
[Heather]
To the tiniest portion of an answer: there's a mosix aware variant of
Knoppix. Check out LWN's Distributions page. (lwn.net/Distributions)
If you get things going, I agree wholeheartedly -- this could make a
really fun article by you about getting your cluster going. Are you
suggesting that you could write this, or that you would like to see
the topic come up?
As such I've changed the gazette@ (main editor) cc to articles@ (article
ideas and submissions)
Currently we have a Red Hat v6 server with sendmail v 8.9.3 running. We have
set a new server with Red Hat 7.3 and sendmail 8.11.6 in order to migrate all
of our services from the current server to the new one.
My question is: May I use the current sendmail.cf and sendmail.mc files in the
new server to avoid manually configuring sendmail again? I mean, to copy such
files to the new server and restart sendmail, do you think it will work fine?
Thank you very much.
[Thomas]
Why have you not tested it yourselves? You certainly have nothing to loose
by doing so
To answer your question though, you will have no problems
using the configuration files. They're based in a rather interesting
scripting language called m4 which is a separate entity to sendmail.
[Sluggo]
He may not know what TAG is, Thomas.
TAG is The Answer Gang, the group of volunteers at Linux Gazette that
answers tech-support questions. All questions and answers are
considered for publication in a future issue. Please address follow-ups or
future questions to tag@lists.linuxgazette.net. More information about
The Answer Gang is at: http://linuxgazette.net/tag/ask-the-gang.html
Thank you all for answering, I really appreciate it.
I would disagree with a softening of the advise never to log in as root.
Respectively, I would remind that there are two types of Linux/Unix
Administrators --
1. Those who have trashed the entire system by mistake.
2. Those who will trash the entire system some time in the future.
Personally, I avoid logging in as root as much as possible, preferring
to su for a brief perioed when necessary.
Kind regards
Tony Dearson
[Sluggo]
I don't see much difference between logging in as root vs su'ing. Root
is root.
[Ben]
Ever dip your finger in liquid nitrogen, Mike? It's a geek thing, FSVO
"geek" (e.g., radars and black-body targets.) In-and-out, no problem.
Keep it there for a full second or so, and you'll have frostbite that
may require amputation. (Five seconds or so, and you'll be able to
shatter it like glass.) When you log in as root, everything you do -
and the chain of consequences that proceeds from it - is done as root,
and you must consider that consequence tree for every single command you
type including "ls". When I type a command, then realize that I need
root privs for it, I just hit the up-arrow, type:
"<Ctrl-A>su -c "
think about the effects, and press 'Enter'. Somehow, typing 10
characters (7 if I know it beforehand) does not seem to me to be the
Sysyphus' burden (with a bit of Prometheus thrown in for effect) that
you insist it is.
[Sluggo]
I hate typing
"su[Enter][password][Enter][command][Enter][ctrl-D]" all the time, or
even worse, "su -c 'command in quotes'[Enter][password][Enter]".
[Ben]
[shrug] Everyone has their pet hates, of course. This does not in any
way correlate to what makes for reasonable system usage.
[Sluggo]
I love
the fact that konsole has a "Root console" menu option, so I can get a
root session any time without using the arcane su syntax.
[Ben]
In what way is it arcane? Is "-c" in some way connected to the Norse
Edda and the Boghaz-keuy Babylonian tablets? I wasn't aware. Molehill ->
mountain requires a sufficient amount of dirt, and I'm afraid there just
isn't enough.
[Sluggo]
(You do have
to type the root password, of course.) Even with that, I usually leave
the session open in the background for a while in case I need it again.
The most important thing I ever learned (from sysadmin Pann McCuaig if
he's reading) is, sit on your hands before pressing [Enter] after
any potentially destructive command like "rm -rf", "rsync", etc.
[Ben]
[grin] I prefer to use my other end to control my hands when using root
privilege, thanks.
[Sluggo]
Make
sure you're the right user, on the right system, and in the right
current directory. (Actually sitting on your hands is optional, but the
metaphor is good so you don't forget it.)
If all that information doesn't appear in your shell
prompt, read the manpage for your shell and set PS1 or PROMPT
accordingly. My favorite prompt strings for zsh are:
Whenever I'm root, my text is all in red. Makes for a decent clue, hard
to miss.
[Sluggo]
(Normally people use '#' to distinguish their root prompt, but I find
that too easy to miss. I also need '$' to show it's bash rather than
zsh (whose conventional symbol is '%').
Another important thing to remember is that "su -" gives you root's (or
anybody's) full shell environment,
[Thomas]
...so that $SHELL, and other ~/.profile
files, etc are sourced, as well as various other exported $VARIABLES are
updated.
[Sluggo]
while "su" alone gives you a partial
environment that doesn't include all their environment vars. E.g.,
"echo $USER" (or "echo $LOGNAME" in some shells) shows your login rather
than root's. That may cause some programs to do the Wrong Thing; e.g.,
'mutt' will read your mail rather than root's.
[Thomas]
Will not change $SHELL, and other variables.
This can be negated with the:
su -m
switch.
[Sluggo]
(No, you still shouldn't
read mail as root anyway! Put those dark classes back in your pocket,
Ben.)
Mike,
That's a good tip, thanks. A small detail I've never come across before
Privately sent to Ben, and forwarded to the Linux Gazette
for publication.
-- Heather
Dear sir,
I am using Red Hat 7.2 and facing same problem after
installing sendmail. Might I have made mistake while
confugering Sendmail. But I am trying to find the
mistake.
I got a lot Encouragement from ur efforts.
Thanking you.
With Regards,
Sam
[Thomas]
Are you saying that you followed Ben's article
(http://linuxgazette.net/issue52/okopnik.html)
and that after you installed sendmail the problem arose that /bin/bash access
denied? That being the case I would either do:
rpm -qilp sendmail
to generate a list of files belonging to that package, and check the perms
for each (assuming you're using an RPM-based distro), or under debian:
dpkg -L sendmail
I am deliberately being vague here.
Your question as it stands lacks structure, information and meaning. I
suggest you look here:
I'm not sure where this should be shoehorned into our FAQs, but - having
just read this excellent 25-page paper, I think it should be required
reading for prospective authors (or anyone who wants to write a
technical paper). "Clarity in Technical Reporting", which had a long run
as an underground publication at NASA, was officially published by The
Powers That Be once they caught on (thus proving that they were worthy
of their positions. All hail, etc.)
Hi all. Just grabbed the #97 and am having some problems reading it on
Mozilla 1.6b on Mandrake 9.0. The navigation box overlays the first
several lines of each article. I played a bit with lg.css (which I don't
know anything about) and got it readable by outcommenting the line:
position: absolute;
at line 53.
Now, the article text overlays the nav. box. Not what you intended, but
it is readable... a real fix would be appreciated. Hey, this bug might
even spur me on to learn how to use stylesheets
And: keep up the great work!
[Mike]
Thanks. Our stylesheet editor, Rob Tougher, is away on family business,
and I have only a rudimentary knowledge of CSS. Are the menu links
showing up one per line with a black bullet left of each? And the
breadcrumbs (=Yahoo bar) too? That was something I specifically fixed
yesterday, because our stylesheet fixes this month (for Netscape 4 and
Phoenix) broke compatibility with issue 97. If you do "View Source" on
an offending page and scroll to the bottom, the menu links should not
have <li>...</li> around them. If they do, your browser is caching an
obsolete version of the page.
I tried your solution but it breaks in my Galeon: it moves the menubar
to the very top, covering part of the logo and Tux. While this is
readable, it's not acceptable. Thanks for letting us know anyway, and
for any other ideas you might have later.
A new problem we have is that after I added the "Contact Us" link, the
menubar is extending too far to the right past Tux. I couldn't figure
out how to prevent that. Do you have any ideas?
Well, problem is solved. I dl'd the version on the site just now and it
works just fine. Guess I grabbed while you were putting, or something.
Thanks!
Been reading for a while and lost the site when you started having problems.
Nice to find it again!
Keep up the good work.
Martin
Thanks, Martin, I'm glad you like how we've kept it. Happy New Year
-- Heather
how to join the TAG mailing list?
Wed, 31 Dec 2003 17:29:51 -0800
sanjaya singharage (sanjayas from opensource.lk)
Answered By Mike Orr (Sluggo), Breen Mullins, Ben Okopnik, Thomas
Adam, Heather Stern
This is a multi-part message in MIME format.
You'd think that if Sanjaya had been on this list before that he would
have remembered that none of us appreciate MIME encoded e-mails.... oh
well...
We couldn't carry the subscribers over from the old linux-questions-only
list because SSC wouldn't give us the member roster. Now the list is
fully under our control.
[Breen]
By the way, I'm back too. It's been pretty busy IRL, but I finally got
around
to moving here.
[Ben]
It's not supposed to be obvious; you were just shown a shortcut. If you
want to join, the what you're really supposed to do is read the TAG
FAQ at:
which will direct you to the subscription address, etc. after explaining
the necessary facts of life - such as, you do not need to subscribe to TAG
in order to ask a question. Subscriptions to TAG are for those who are
willing and able to answer Linux questions.
Good point - we should update the "About TAG" blurb on the mailman link
given above to include this concept, give the hotlink to member policy,
and advise querents to read ask-the-gang and then just send mail.
-- Heather
This page edited and maintained by the Editors of Linux Gazette HTML script maintained by Heather Stern of Starshine Technical Services, http://www.starshine.org/
Many new laptops come without a floppy drive; the one that I just
bought, an Acer Aspire 2003LMi, does not have one - it's available as an
extra option, but I don't see myself needing it. However, part of my
standard method for converting Win-machines to dual-boot involves using
FIPS, which I use to "shrink" the Wind0ws partition to a minimal size -
and FIPS normally runs from a floppy. What to do?
Here's an interesting fact that many people may not be aware of: the
bootable part of a CD consists of nothing more than a bootable floppy
image. So, I simply took a DOS boot floppy containing FIPS, and made a
byte-by-byte copy:
# Create a directory to hold the CD data (none at the moment) plus "boot"
ben@Fenrir:~$ mkdir -p /tmp/cdrom/boot
# Create byte-by-byte copy in "boot.img"
ben@Fenrir:~$ dd if=/dev/fd0 of=/tmp/cdrom/boot/boot.img
I then created an ISO image containing that disk copy. If I wanted any
other data on that CD (I'll probably make another one with a bunch of
DOS utilities on it later; I've been using bootable DOS "tool" floppies
for over 20 years to repair broken Wind0ws systems), I'd copy that data
into "/tmp/cdrom", and it would become part of that image.
The only downside to this is not being able to save the boot sector to
the floppy before repartitioning... but in the worst case, it's a new
system without any of my data on it, and it's not a concern. Besides, I
have Linux, and boot sector recovery is rather trivial.
Nice value in XFree86 startup
Sat, 24 Jan 2004 21:56:43 +0000
Thomas Adam (The LG Weekend Mechanic)
Question by Jacobo ;-) (jacobo221 from hotmail.com)
Hi,
first of all, sorry for my english, I try my best, but I use to fail in
spelling and grammar
Well, I'm writting to you because I have a problem at Xfree86's startup. I'm
useing kernel 2.4, so I have, in Xwrapper.config, nice value set to -10
("nice_value=-10"), but when entering "startx", just before entering
Xfree86, a message is shown: "warning: process set to nice value -11 instead
of -10 as requested". If I then change nice value to -11, then the startx
script changes nice value to -12. If I change it to -12, it changes it to
-13, and so on for any number between [-20, 19]. I've searched for many days
(weeks, in fact) on the web and asked at #debian, but no answers. I hope you
can lend me a hand. Btw, XFree86 runs perfectly ok, it's just that I hate
having error messages.
Thanx a lot for your help!
[Thomas]
By my knowledge of how X starts up it is not startx which is changing
this. Just out of curiosity, run (as root):
dpkg-reconfigure xserver-common
to see if that fixes anything. I have grepped through the startup files
that I use (I am running debian unstable) and there is nothing besides the
value in /etc/X11/Xwrapper.config which sets or changes the nice value of
X.
As a long shot (and possibly a complete aside) you don't have the "and"
package installed, do you?
CPU Support on Linux
Sun, 22 Jun 2003 14:10:02 +0100 (BST)
Ashwin N (The LG Answer Gang)
Question by Shishir_Bagchi (Shishir_Bagchi from lko.tcs.co.in)
How do I get an E-maikl to above subject? They interrupt my messages.
[Jason]
What you seem to be asking is "why am I getting messages from Mailer-Daemon"?
A message from "Mailer-Daemon" is probably the MTA (Mail transport agent: The
software that delivers mail.) sending a message that says, in effect, "excuse
me, but you seem to be confused. There is no such mailbox here."
claim no such addresses exist, etc.,
[Jason]
The automated message you're recieving means just what it says: No such
address exists. For example, if you try to send email to bob@example.com, and
there is no user bob at example.com, you're probably going to get a bounce
message that says no such address exists.
yet they cannot be questioned or challenged ever when they are wrong. Help.
[Jason]
The reason "they" cannot be questioned is because you're getting an automated
message: It was not send by a human.
Nobody would know better whether an address exists than the system you're
sending mail to, so I would say that the system you're sending mail to is
right and you are wrong.
what is "no"? (missing prog for binutils compile)
Thu, 29 May 2003 15:53:14 -0700
Faber fedor, Jim Dennis, Alan Mudra (The LG Answer Gang)
Question by Karl-Heinz Herrmann
Now -- it's an unbearable situation that my Linux doesn't know "no".... But
I've no idea what it is and you can imagine that a google for "no" even with
linux and some other keywords around are not very helpful.
Any ideas?
[Faber]
Maybe it's called "nein" on your computer? <grvf>
[JimD]
I think /usr/bin/no was (would be) a counterpart to the old
/usr/bin/yes command:
#!/bin/sh
OUTPUT='y'
[ "$#" -gt 1] && OUTPUT="$*"
while : ; do
echo "$OUTPUT"
done
... so "no" could just be an alias or script that calls /usr/bin/yes
with the "no" argument:
/usr/bin/yes no
I realize this sounds silly and stupid, and April 1st is long past for
this year. But I'm not kidding. That Makefile (or whatever) seems to
actually want to pipe an endless stream of "n" or "no" lines into some
other process. (/usr/bin/yes was traditionally used in a pipeline with
fsck to automate the process of repairing a filesystem that need lots
of work -- then they just added the -y option to the GNU/Linux versions
of fsck.
I put the question up with bugreports for binutils and got:
[Alan]
> /bin/sh: no: command not found
This is a result of binutils being stuck on using old buggy autoconf.
Install a new version of GNU gettext, or configure with --disable-nls.
--
Alan Modra
IBM OzLabs - Linux Technology Centre
unfortunately with a new gettext (gettext-0.12.1.tar.gz) and nls enabled I
get a linker error for some gettext symbol. The solution without nls works
for getting binutils compiled. I try that on the 2.5.70 kernel sometime soon.
Does anybody know of a backport to 2.4.X of the preempt patch and or the I/O
scheme patches mentioned on kerneltrap right now? I'm not yet sure what else
will break if I switch to 2.5.X. At least NVdriver, lt_serial+lt_modem and
vmware kernel modules would be nice to have.
What is Linux Torvalds wife famous for
Thu, 19 Jun 2003 15:00:53 +0100 (BST)
Ashwin N, Thomas Adam (The LG Answer Gang)
Question by Pam Drake (pdrake from northern.wvnet.edu)
[Thomas]
Well, she's famous for being married to Linus Torvalds.
[Ashwin]
She is also famous for being Finland's champion in kung-fu or some similar
martial art
Linus was interviewed in Issue 67 of Linux Journal, by Marjorie
Richardson at the Linux World Expo of that year. Tove was 6 years
running, the Finnish champion for karate, specializing in precise Kata
(the forms), then moved on to other interests.
-- Heather
pppd problem
Thu, 26 Jun 2003 01:30:57 +0530
Joydeep Bakshi (joy12 from vsnl.net)
Answered By Thomas Adam
Hi all,
I use kppp under linux to dial-up my isp. but there is a strange problem
happens with me. when I dial-up from windows98 it connects with the isp at
the very first attempt. but under Linux (debian woody) kppp takes at-least
3/4 attempts to connect the isp, and during the failure it shows *pppd can't
be started * . I have also started pppd from root manually at the time of
hooking, but the result is same. could some one please suggest me how to fix
the problem ?
thanks in advanced.
[Thomas]
You need to ensure that you have a ppp-chat script enabled which is used
by pppd to comminicate with the modem to send certain signals, etc.
wvdial will help for this
PPP & ETH won't work concurrently
Fri, 5 Dec 2003 09:48:52 -0800 (PST)
Joe Slobotnik (cj from rt.nl)
Answered By Thomas Adam, Karl-Heinz Herrmann
Hi,
I'm running red hat 9 with an ethernet card to a LAN and a modem for dial up. I'm using kppp for the dial up. When the eth0 int is active, kppp will establish a connection but DNS fails. The DNS listed for both interfaces is the dialup one, but I can't resolve names unless the eth0 is shutdown. How can I , an ordinary joe get these things to work at the same time ?
thanks,
cj
[Thomas]
There is either a really easy or hard explanation. I think I am right in
saying that you need to have a correct entry in your routing table to use
the two concurrently, since the routing will not know.
As far as DNS goes, do you have a valid entry in /etc/resolv.conf and also
an entry in /etc/nsswitch.conf:
hosts: files dns
like that? If not, add the "dns" after the word "files".
[K.-H.]
Well kppp (or pppd which kppp calls) refuses to setup a default route if
one already exists. The magic scrying ball (glass?) would suggest with
eth0 up you've a default route set. On dailup via kppp you won't get a
default route to your ppp0 interface and therefore DNS lookups to the
world outside never reach there.
This is how it should look lie with both eth0 and ppp0 (kppp) up (I cut
out three columns which are unimportant):
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Iface
62.104.218.38 0.0.0.0 255.255.255.255 UH ppp0
192.168.2.3 0.0.0.0 255.255.255.255 UH dummy0
10.10.10.0 0.0.0.0 255.255.255.0 U eth0
172.16.57.0 0.0.0.0 255.255.255.0 U vmnet1
0.0.0.0 62.104.218.38 0.0.0.0 UG ppp0
I've two private networks, real ethernet as 10.10.10.0 and
vmware virtual host as 172.16.57.0. The "0" at the end suggests and a
genmask of 255.255.255.0 proves that these are networks, i.e. not a
single host but all hosts 10.10.10.x with 1<x<255 (zero is broadcast
address to all). If you look at the flags "H" means this is only one
single host, "G" means this is a gateway.
The last line is the default routing, i.e. if no other routing rule
applies all remaining traffic this way. The way is interface ppp0 and
the target the gateway IP 62.104.218.38, our remote host on the other
end of the modem line (see first line).
Now I'm pretty sure that in your case there is a line like:
0.0.0.0 [some IP] 0.0.0.0 UG eth0
If this is the case kppp will not touch it and if you would have looked
carefully in your /var/log/messages (or kppp's log) you would have found
an error telling you this.
If I guessed right run (as root):
route del default
then start kppp's dialin
After that figure out why RedHat sets a default route (I'm with the
lizard and without hats).
If I did not guess right you oviously didn't give enough information....
I am having a peculiar problem with the Red Hat 9 installation. I am trying
to install it on a IBM thinkpad 1721 laptop with a formatted hard drive. The
laptop has a combo floppy/CD ROM drive. The laptop seems to be seeing the CD
ROM dirve but it is not able to read anything off it. The laptop has a
PCMCIA network adapter card which I used to connect to my home networking.
The other laptop on the home networking has Win 2000 running on it. Here are
the following waya I tried to install Linux -
1. I created a Linux boot disk using the pcmcia.img and booted the IBM
laptop and selected NFS install. On the other laptop I copied all the Red
Hat folders from the 3 installation CDs. I selected "Automatic DNS
configuration" to configure the DNS names for the IBM machine. For the NFS
server name I typed in the name (which did not have a domain name since I
log into a workgroup and not a domain) and typed in the actual directory of
the Redhat parent folder (d:\redhat9). But I get an error that the drive
could not be mounted.
Then I grabbed a Freeware called NFSClientServer and installed it on the Win
200 laptop. Here I exported the RedHat directory (d:\RedHat9\RedHat). Then I
followed the above steps on the IBM machine for NFS installation. Even
though my requests form the IBM machine seemed to log in the Server log on
the Win 2000 machine, I still could not mount the directory. What am I
missing?
2. Failing the above attempt, I attached a SCSI CD ROM to my IBM machine.
Then I made a boot disk using the bootdisk.img. After booting the IBM, I
selected "local CDROM" for the source, but I kept getting the message "No
software found in CDROM". Obvously, Linux was not seeing my SCSI CDROM, but
since it detected my internal (failed) CDROM, and could not read off of it,
it was giving me the message. How do I make it look at my SCSI CDROM?
I even tried making a boot disk using the "drvblock.img" file, but for some
reason when I use the disk, I get the message " No operation system found".
When I look at the disk contents the, it seems like the format information
on the disk is lost and Windows explorer asks me if I "want to format the
disk"? I even tried the same using Linux 7.3 (Red Hat), same error.
Please help.
Regards,
A Wexed Linux Installer.
[Mike]
OK - your best bet would be a HD install.
If you have copied the folders over you have enough HD space.
In w2k make a directory to hold the iso images
In turn copy the cds to this directory (not the contents - I think
Nero should be able to do this)
You should then have three files in this directory, all ending in
.iso
Now boot using your boot.img disk.
When you get get prompted for installation type, choose HD
then select the right partition where there are held (probably
/dev/hda1 on your system if you choose the first partition)
then select the directory you have placed the .iso files in.
Then you should be good to go.
pam and ssh
Wed, 25 Jun 2003 10:08:28 -0500
Lou Lohman (LLohman from WeberMarking.com)
Answered By Thomas Adam, Michael Gargiullo
Dear Answer Gang,
Try as I might, I am unable to figure out just what to do to allow a trusted
user on a trusted host to use 'scp' to copy files from one system to another
WITHOUT a password prompt. We WANT to do this in order to use 'scp' in
scripts initiated by CRON. We NEED to do this because 'rcp' gives us 'file
too large' responses. I have tried modifying /etc/pam.d/sshd (at least to
the limits of my understanding). I can make scp NOT work at all, or require
the password, but I cannot make it work without the password. Changing the
/etc/ssh/sshd_config file doesn't make any difference, that I can see, even
though that file now says 'go ahead and use the rhosts files'.
Can someone help me? Anyone? I would offer a carrot, like "I'll subscribe
to LJ", but I already do. I would offer virtual hugs and kisses, but the
respondent might be male, in which case an old homophobe like me is going to
have a problem. How about "undying gratitude"? Yeah, that's the ticket ...
it's easy, it's cheap, I could even teach my kids to sing appropriate
praises .. yeah, that's it .... Where's Jon Lovitz when you really need him?
Lou Lohman
Don't just BELIEVE!! Consume information like a starving person, and then sort it out for yourself.
And we know he reads ask-the-gang.html, he gave us explicit permission
to publish the whole thread just like we ask for
Now if we could get
people to remember to turn off HTML in their email...
-- Heather
[Thomas]
You need to run "scp" with the "-B" flag, ie.
scp -B files thomas@thomas
Batchmode doesn't require password authentication. You can also add this
in /etc/ssh_config as...
BatchMode=yes
that way, you don't have to pass the -B switch each time.
[Mike]
You can also exchange keys from one user/machine to the other. If you
want to copy from machine A to machine B. On machine A, as the user that
needs to copy, run ssh-keygen -t dsa. This will generate the key pair.
Then he'll need to copy the contents of ~/.ssh/id_dsa.pub from machine A
into ~/.ssh/authorized_keys on machine B. Then copying from machine A
to machine B won't require a password for this user. Do the reverse to
copy from B to A.
I have some problems trying to connect with my HSP Pctel Micromodem 56.I have
RedHat 9 installed and I use kppp for conneting to the net.When I try to
connect the handshake goes on well and all of a sudden I get this "No
Carrier" message.Can you tell what causes this error and what is the
solution.............please sugget an alternative driver if available.At
present I am using the Pctel drivers available at www.linmodems.org version
0.9.6 . Can any special AT command string help?????? If yes please suggest a
solution.Or using any other dialer will help????
Thanks in advance.
Vivek.
[K.-H.]
no carrier means the modem is unable to detect the carrier frequency onto
which data would be encoded. As this is a fatal problem it hangs up and tells
you "no carrier".
Why is the carrier gone? difficult to tell from here. What do your logfiles
say? kppp has a log button, use it!
At exactly what point of connection negotiation does the error occur? Do
you get the "connect" in the log window? Then the dialing is finished and
control passed to pppd -- which logs it's messages in /var/log/messages or
some such place. Go look for it. You can pass additional option to pppd in
kppp: add "debug". Make sure to press the add button in the kppp window so
the new option is actually used (should show up in the lower larger window).
My guess: serial connection gets established, pppd get's into some trouble
negotiating the ppp parameters (user/passwd? pap <-> chap <-> terminal
authentification, compression,.... ) and the other side terminates on you.
Your modem detects the lost carrier and tells you.
Here's a suggestion for your 2 cent tips section - I hope you like it.
When working with Linux servers I don't run X, preferring to make use
of the virtual terminals via Alt-F1, Alt-F2, Alt-F3 etc. The problem is
knowing which screen I am looking at, so I have devised a custom
prompt by including the following two lines in my .profile:
TTY=`basename \`tty\"
export PS1="[$TTY] \w$ "
Cool
The first line sets the environment variable TTY to the number of the
current virtual terminal, the second sets my prompt to show that number
and the current working directory as in this example:
[2] /etc/xinetd.d$
That's it!
Toby Poynder
London, UK
I must admit that I often find just typing in "tty" is more efficient than
having it set in one's $PS1 prompt.
-- Thomas
workaround for black lines on Radeon mobility 7500
If that isn't enough to get you to run out and download it, you're probably
not the kind of person who would enjoy it.
[Ben]
And don't forget to take a look at the sample fractals that come with
the "xaos" package, located (under Debian) in /usr/share/XaoS/examples.
(I've got all of these converted to JPGs and use them as
randomly-selected desktop backgrounds.) Beautiful.
[Faber]
Yes, a very nice toy, but I still miss Fractint for DOS. The X/Windows
version had all the features, but just doesn't cut it somehow. A
Mandelbrot set not being full screen looses something, not to mention
they're slower than the DOS version.
I also had an internal IBM program[1] that would map out different phase
space maps stereographically, looking like some of the maps in the
Discrete screen saver in xscreensaver.. Man that was a fun toy; I wish I
could find the source code (it was floating around here on disk years
ago); I'd try to port that to Linux.
[1] For those of you that don't know it, IBM has a a mini Internet in
place with their own version of "free software", to wit programs written
by Beamers and distributed to other Beamers for fun and collective
profit. And since some of those people had multiple PhDs and worked on
esoteric stuff, some interesting programs cropped up. It's the only
thing I miss from my IBM days.
This page edited and maintained by the Editors of Linux Gazette HTML script maintained by Heather Stern of Starshine Technical Services, http://www.starshine.org/
The Answer Gang
Linux Gazette 99: The Answer Gang (TWDT)The Answer Gang 99:
Greetings, gentle readers, and welcome once more to the world of
the Answer Gang. TAG, we're it
I have to confess that I wondered to myself, what should I babble
about this time? As I look at the back issues, I notice some
interesting statistics... apparently, I shouldn't feel surprised
that things here at Linux Gazette are a bit hectic. That's
consistent with all our past Februaries. Ain't tradition grand?
February here where I sit, is often considered the month of romance.
I have to admit... I (heart) Linux Gazette.
I mean, it's not like I will be buying it chocolates or wondering what
its favorite cologne is. But I put a bunch of work in every month - I
really like knowing you people out there are reading (hint, hint;
tell us what you like in this
stuff)! I love seeing what sort of curious troubles are out there to solve
- the most curious, the kinds of things whose answers change over time.
And it's always worth seeing what cool answers are out there.
You don't have to actually be a member of the Answer Gang to send us juicy
answers, either. An honorary Answerbubble to folks who send us their good
stuff - or cc us on the the juicy tip they are sending to someone anyway.
When we got the Gang together, the heading mentioned a few of the active
posters... and you. That hasn't changed. It's teamwork that makes this
all happen, and I'm proud to be here. Thomas and I have teamed up to bring
you the juiciest threads this month, and I hope they make your time with
Linux this month - "just a little more fun!"
Backing up with tar
From Tom Brown
Answered By: Thomas Adam, Chaz Peters, Karl-Heinz Herrmann, Ben Okopnik, Robos
I'm trying to backup my Linux installation with tar, using a second
hard-drive in my system, rather than a tape-drive or CD. The trouble is,
I have a 2-GB file size limit on the destination (It's Fat32, so I can
also use it for Windows backups), so I have to do it in a lot of little
chunks (even with compression). Is there another solution to this,
either a fancy shell script, awk script, or some combination of tar
options that would produce the multiple destination files I'm looking
for? If I keep doing it manually like I am now, I know I'll never
maintain an up-to-date system backup like I should. I've found tape
options for tar that control multivolume backups, and tape length, but
nothing for multiple files.
[Chaz]
Backups can be a pain, especially ones that require manual operation. I
like to automate them as much as possible. The following is a script I
made for Kathy's Debian machine. Usually I prefer to backup over a
network to another machine, however she has dialup and no other machines
on a LAN. I use rsync because it's fast and works well. rsync is a file
transfer program capable of efficient remote update via a fast
differencing algorithm. This program is run once a week via cron, it
works very well for hassle free automated backups as long as you have
enough disk space. If you require compression, this is not what you want.
I do not recommend using compression for backups, compression reduces the
chances that the data will be recoverable.
tar zcvf /windows/s/suse/back_tbrown.tgz /home/tbrown
Oh, I tried the Suse backup/restore function, and could not restore the
resulting files. The .tar.gz files within the .tar archives (don't know
why they did it that way) seem to be corrupted. So, I figured I'd do it
myself.
[Chaz]
SuSE, sorry the dpkg part of my script won't help...
Note the script lacks a secondary archive, that could be disastrous in a
few cases. We do have an older backup on CDR and at some point I would
like to transfer it to a laptop or something for other more recent off
site copies. She can also selectivily transfer files via dialup so that I
can back them up.
When I get more disk space, I am going to look into using better archival
techniques. I have heard good things about Dirvish, a fast, disk based,
rotating network backup system. A dirvish backup vault is like a time
machine for your data. http://www.pegasys.ws/dirvish
[Thomas]
What you can do is something like this:
(cd /src/dir && tar cf - . ) | (cd /dest/dir && tar xvfp -)
where /src/dir is the directory you're starting from, and /dest/dir is the
final destination that the files (dir's) will end up to.
Since you say that this is going to a FAT32 volume, that will not preserve
file permissions. The only way you can achieve that is by making a tar
file.
Thanks. That's why I didn't just cp the directories over.
[Thomas]
Your other option is to make an archive and burn it to CD. One thing you
might want to try though, is you are going to make a tar archive, run it
with the "j" flag when you create it. That'll use bzip2 and might compact
some more space.
[K.-H.]
You might have a look at afio instead of tar. It's more robust against
data errors in the archives then tar and from reading the manpage I'm
not quite sure if you can specify archive-filenames which are
automatically numbered for multivolume. If not you can still automate
things with the "promptscript" option. You archive to a specific dummy
file, the script will mv/rename it to something useful and (number,
date,...) and continue. To get rid of the prompts (or answer them
automatically) should not be that difficult.
Be careful to read the basics: afio wants a list of files to be archived
piped in on STDIN.
This might be a good startingpoint (no multivolumes, add that yourself):
[Ben]
Make your giant tarball, then use the 'split' utility to break it up
into chunks. When you're ready to use it, just 'cat' all the pieces in
order (which is how they'll be named by 'split') into a single file that
you can untar. As someone mentioned, 'j' rather than 'z' gives you even
better compression on large files.
[Heather] Since j invokes bzip2 compression, yes. I wouldn't use it if anything
needs to be unpacked on a non-linux system though; other OS' are shabby
at bzip2 support.
That would work fine, except that the tarball is too big to be created
on the destination file system in the first place. What I'm looking for
is some way of creating a lot of smaller tarballs right from the start.
[Ben]
What I meant was to create it on the "source" system, not the "target"
one, then split and transfer. However, you can do it "in flight", too:
tar cvzf - * | split -b 100k backup-01-15-04
[Thomas]
Since the destination is not a Unix system, the use of the "-p" flag to
preserve permissions is a must in this instance.
[Ben]
It's not really relevant to the host OS; the permissions that matter are
"inside" the tarball. However, you're right anyway - in a backup,
[Thomas]
Indeed.
[Ben]
permissions should be preserved, and I lost track of that in generating
a random example of "split" usage. In fact, for backups, the "tar"
string should be:
tar cvzpSf - *
(add sparse file handling, as well.)
[Thomas]
LOL, I don't know, Ben.... all that Yoga and the like is going to your
head, just make sure you:
tar cvzpSf
your linux knowledge
I for one, would be very interested in that tarball...
[Ben]
Sorry, even the pieces would be too large to fit on any possible host
system. Although there's a lot of sparse files there, too.
[Ben]
This will create a load of 100k-sized files called "backup-01-15-04aa",
"backup-01-15-04ab", etc. If the destination was a Unix system, I'd
suggest piping "tar" into SSH, catching it on the far end and then_
splitting it - all done in one shot.
[Robos]
I'd rather use netcat instead of ssh. Depending on the connection certainly
(didn't read all). But ssh adds quite a load more to the already busy cpu
which tries to to bzip compression on the fly...
I'd love to find out why the Suse backup tarballs won't untar, since
Yast2 appears to do the kind of backup I want. I'm overlooking something
there, I just know it, since the feature wouldn't exist in Suse if it
didn't work.
[Ben]
Don't know anything about SuSE backup, but the above should do what you
want.
Hunting for new desktop hardware
From Karl-Heinz Herrmann
[Heather] The question this time is from our own Karl-Heinz Herrmann. When you
play on the edge, it's best to ask a few buddies to keep an eye out for
the loose rocks...
Answered By: Robos, Rick Moen, Heather Stern
Hi Tags,
I'm hunting for new desktop hardware and getting confused and somewhat
frustrated.
[Robos]
Me too (sort of)
The best German computer magzine (c't) has of course plenty
of tests and recommendations as well as "optimal PCs" for certain usage
profiles. My problem is, I want a decent performing raid 5 (burned to
often by sudden drive failure).
[Robos]
Have that already in the gangway - working flawlessly for several years now.
80GB raid 5.
[Rick]
Do you expect to be so incredibly short of disk space that RAID1 isn't
an option?
[Heather]
Funny you should mention that; I met someone recently who got burned by
sudden drive failure on a RAID 1 system - and to their horror, while the
other drive was fine contents-wise, it was also in a state where it
couldn't be brought up normally to be looked at - expected a twin to be
present, and nobody wanted to risk it considering a "mew" drive the
master and wiping it. It needed a visit to the drive recovery shop.
RAID is not a panacea - a cure-all - against hardware failure; it merely
helps. Usually, a lot. But it's specifically not a solution against
slow corruption via software failure. Anyone working with oversized
disks should seriously consider their backup-and-recovery plans too.
no -- raid1 would be an option. If some 50GB actual space are there that
should be enough for some time, (temp) video data won't need raid1.
With recent consumer boards I seem to have the option of using IDE
drives, sure. Cheap huge drive space and all. If I understand the board
details right all of the regular ATA interfaces are on the PCI bus -- so
a raid with 3 or 4 drives there is hogging the PCI bus.
There are SATA interfaces which are not going through the PCI
interface but have a separate one -- just Linux seems not to happy with
SATA yet and most boards have 2 SATA drives -- not enough for raid5. If
I could be sure that a raid1 is running stable and at full speed in
Linux with SATA drives hat might be an optin, as SATA brings plenty of
disk space for a reasonable price).
[Rick]
Huh? RAID1 will inherently be faster than RAID5. But either is fine
with the md driver.
That's not what I meant -- from all I heard SATA is still highly
experimental and often not supported at all. Most often I heard as
problemsolver to switch them to IDE.
[Rick Moen]
That's one workaround, of three total.
I'm not quite sure if the SATA
drives are connected to the IDE connections for that or if the SATA
connections can be used in a compatibility mode.
[Rick]
The latter -- where the BIOS supports doing so.
And I also don't know
what performance changes that might bring along.
[Rick]
Who cares? You use the drive in that mode only long enough to compile a
new kernel with the requisite support for the SATA chipset in question.
I've been googling quite some time again and slowly I get an idea on
Linux support. libata is providing sata support --
[Rick]
More precisely, libata is providing better and broader SATA support.
patch in 2.4,
already included in 2.6 but still under heavy development as the latest
changelogs and bugfixes show.
[Rick]
Correct. However, there is non-libata support for several SATA chipsets
in recent 2.4 kernels. (Of those, 3Ware Escalade is supported for a
long ways back in 2.4.x.)
I'm not sure about the Via K8T800 chipset
[Rick]
That isn't a SATA chipset. It's a motherboard chipset -- always
accompanied, to the best of my knowledge, by the VIA VT8237 South
Bridge, which does SATA and is supported in libata.
but the nforce3 seems mostly supported (sound seems a dead weight).
[Rick]
Nvidia nforce3 is also not an SATA chipset, but rather a motherboard
one -- always accompanied, to the best of my knowledge, by a really
awful Promise R20378 SATA / software-RAID chip. The sound is indeed
provided by a really dreadful Realtek ALC650 chip.
I would run, not walk, in just about any direction leading away from
that kit.
There are driver tar packages from nvidia for download (at least partly
source, not sure if some binary only parts are in there too). The Sata
controller on nforce3 boards seem to be Silicon Image like Sil3112A,
Sil3114.
Robos: You said they are supported, do you have pointers for details?
[Rick]
That is conceivable, but is not what I've been seeing. It's possible
that it's dubious-quality Silicon Image chips in some cases, and awful
quality Promise chips in others.
[Robos]
Yes, my flatmates experience
But he's on holiday so I can't ask him. And
I don't really recall any details.
[Heather]
I've got one; a buddy just compiled 2.6.1-mm4 and with a couple of
stumbles making sure other parts work also, is now ready to fall the
rest of the way in love with his Athlon/SATA system, bearing an
sil3114 chipset. Statistics on exactly how cool not available just
yet
Has anyboy here SATA drives working? Performance? kernel versions? any
patches? And on what particular chipset?
[Rick]
Best quality: Adaptec 24x0, 3Ware Escalade -- in that order. But you
won't find those built into motherboards. Usually, what you find inside
motherboard chipsets is Intel ICH5. Which is basically OK.
I don't mind AMD/P4 or even
AMD64 (new 3000+ is reasonably priced).
[Robos]
OK, I don't have sata drives but I read something about it. It seems (see
latest c't) that all the upcoming athlon 64 boards have sata (since it's in
their north|southbride) and quite a lot of them already have hardware raid0
and 1.
Yes -- the hardware is there -- but as far as I know none of the
hardware raids are supported in Linux, and booting Linux off SATA drives
which run actually as SATA and not in some IDE-emulation mode is
difficult. What I do not know if once a Lniux kernel takes over if they
are running as real SATA? Performance penalty for IDE emulation mode?
And the boards tested in latest c't all have two SATA connectors, i.e.
to few for raid5. Of course I could plug in 1 or 2 additional SCSI
disks on the PCI bus with them.
Alternatively I've already plenty of SCSI hardware in the old one, so I
would maybe even prefer a decent SCSI setup -- but I can't find a
chipset with onboard SCSI controllers -- and if I plug PCI cards I'm
again hogging the PCI bus.
Are there any other chipsets besides nforce2/3, i875,....?
[Rick]
Sure.
Gods above, Karl-Heinz, what sort of incredibly disk-intensive operations
are you planning to conduct?
Well -- right now I've some 32% CPU at full whopping 9MB/s (bonnie 8892
kB/s 31.8 %CPU) from the SCSI raid5 in a Pent. classic board running a
K6II-366MHz. Since I am hunting for new hardware I am thinking of
getting something decent which I (again) can keep for some time. The old
system is running since 1997 after all, with occasional low cost
upgrades(like the K6).
My original idea was: Ok upgrade is necessary, IDE is still not what I
want (performance trouble with my existing Lap and a DVD burner in the
desktop put me off; that cdrecord mess, ide-scsi or not, DMA only with
the right block size,...). So how about some new SCSI drives (3 for
raid5) and a nice new card (U160 one channel should do as I still have
a 2940UW for CD, scanner, etc.) and be done. Well-- the lowest
grade (new) SCSI drives I can locate are some 10k rpm 18 or 36GB drives
which sustain easily 30+MB/s and peak at 70 MB/s. A PCI bus is getting
very much the limiting factor as there are probably more cards in there
as well and I'm looking for a way to avoid building a bottleneck into
the new thing right from the start. Thats why I tried to ask for info's
on boards/chipsets which might have SCSI onboard, preferrably
circumventing the PCI bus.
But you are probably right, just ignore the bottleneck and let 100MB/s
be good enough. In a year the SATA support will be there and offers a
non-PCI extension anyway if needed.
[Rick]
The SATA 1.0 spec claims a 150MB/sec limit on bus transfers. Of course,
the fastest ATA hard drive manufactured today can, under contrived,
artificially ideal conditions, read data at maybe 50 MB/sec, so
saturating ATA/100 or ATA/133 sounds very farfetched for a long time to
come. (Remember, only one ATA device per bus can be active at any
specific time. SATA changes this only by permitting connection of
only one ATA device on the chain.
Aren't there
any "server-boards" with other specifications? In the price lists here
none show up, but a recent acer ad mentioned some "ServerWorks Chipset
GC-SL".
[Robos]
IIRC Silicon Image chips were good under linux. One of the kernel developers
even managed to persuade some hw producer to donate some driver to gpl.
Can't recall which one that was, might have been promise.
Any boards/manufacturers using that chipsets (available in Germany)?
Can anybody point me to details what this chipset can do
(http://www.acer.com doesn't)? Or knows of a nice one (like onboard SCSI [raid]
controller bypassing PCI, don't forget the Linux support)? Or at least
64 bit PCI? All the consumer boards showing up in a A4 page (tiny print)
of boards seem not to have anything special in that regard.
Am I overemphasising the PCI-hogging? To cite above mentioned c't -- If
you run a Gigabit ethernet on PCI it basically saturates the PCI bus.
[Rick]
1. Yes.
Hmm....
[Rick]
2. Using Linux software RAID (md driver) for redundancy, you typically
won't even notice the load relative to (say) a hugely expensive Mylex
SCSI hardware RAID adapter except during RAID volume rebuilds (if you
ever have to do that). In such a case, expect the system to very, very
seriously bog down during the rebuild. With the Mylex card, all of that
would be handled entirely within the array, with basically no system
loading.
Every unclean reboot -- but the raid5 regeneration isn't that bad,
even on the old system.
[Rick]
I'm talking about when one replaces a failed ATA drive in the array.
System will be usable but heavily loaded.
Takes some 10 minutes each 3-5 GB mdX. System
is still usable during that time. Rebuilding is on a low prioritiy.
I picked up another thing while reading through kerneltrap mailinglists:
It seems most onboard so called IDE raid cards are actually just fancy
labelled IDE adapters where the windows driver is nothing but a
software raid.
[Rick]
Yes. Avoid. Use the md driver, instead.
So raid with IDE raid cards are still software raid
unless its the (rather expensive?) real thing.
[Rick]
Yes. "Real thing" basically means 3Ware Escalade 850x or Adaptec AAR
24x0.
Promise was explicitly mentioned.
What I'm not sure is if this is for all raid modes or just raid 5.
[Rick]
Please see my file. You can sometimes get RAID0 disk-spanning using a
BIOS trick.
Simply doubling the writes for a raid 1 can't be that demanding
in card logic after all.... Does anybody know more on this? Has anybody
used IDE raid controllers? How do they behave?
[Rick]
3. Indeed, gigabit ethernet will strain practically any system, even
with jumbo MTU sizes set, and even with a suitable switch and other
associated equipment.
So that's true -- but then I've no immediate use for a Gbit ethernet
now but I want to keep the option open. I think I'll watch the
AMD64 scene for some more time, especially if hardware drivers for
something like TV cards will run with 64bit OS.
If all this bleeding edge driver location stuff is getting sufficiently
annoying I might even grab some PIII off ebay and just plug my existing
SCSI chains in (and swap some 9GB drives in for the 2 and 4.5 one)
Still -- specs on that AMD64 sound nice.
[Rick]
Yes, I certainly am with you, there! But I want to jump onto the 2nd
generation of Opterons, not the first.
There might be a fair bit of video processing in the near future (as
well as the use as home media center with timeshifting and all that
stuff) so there will be incoming video streams on the PCI as well.
[Robos]
we use a 400Mhz Celi for the (software) raid 5 and it works like a charm.
Sometimes my flat-mate and I both stream stuff from there and no clogging or
anything.
Regarding timeshift: if you use dvb you can easily capture the stuff since
it's already in mpeg2 format. No processor usage then. And any recent
platter should be able to both record and play mpeg2 streams simultaneously.
I am toying with the idea of buying an athlon 3000+ soon: no intel (they
have a too high profit margin for my liking), very performant, rather cheap,
64bit (for what it's worth) which linux (debian IIRC) supports rather good.
But stay away from the vobis offering, that sucks
That AMD3000+ is intriguing -- what a small c't article can do to ones
minds
[Robos]
There was one before, like 5 numbers or so, which dealt with 64bit again 32
bit. You might want to read that before too.
Software suspend troubles
From Ben Okopnik
Answered By: Thomas Adam, Kapil Hari Paranjape
So, I've installed swsusp - two patches over the kernel, flip a couple
of switches in the kernel config, and "echo > /proc/swsusp/activate".
Hurrah! Cute little screen with a progress bar, etc., and off goes the
laptop. Now comes the interesting part: when I hit the power button, I
see the Acer splash screen, the LILO message, my cute little Tux, and -
the usual kernel load messages flip over to the 'restore the suspend'
screen with its progress bar... until it gets to the point where it says
"Freezing processes: Freezing remaining tasks." Half a second later, it
restarts the restore process, until it gets to the above message... and
so on, and so on, and so on. PLOKTA didn't do anything - until the next
cycle, where the kernel crashed with a panic (!). I ended up having to
reboot using another kernel - lucky that I always have at least two,
since I don't trust the newly-compiled ones to always work!
There's no info about this on the Web, at least according to Google;
just two hits on the relevant line in the code. I'll have to send some
email to the "swsusp" list and see if they can help.
[Thomas]
How are you actually suspending it though, Ben? I suspend this way:
echo 1 > /proc/acpi/sleep
Huh? Then you're not using "swsusp"'s hibernation feature, but the ACPI
"sleep" instead. It's ">/proc/swsusp/activate", as shown in the swsusp
HOWTO. Incidentally, the ACPI "sleep" doesn't work for me, whether 'echo
1' or 'echo 4'.
Yes, you are quite right, the two are different. I really
am not sure what to suggest other than ask on the swsusp ML. I am not
trying to fob you off or anything, but this kind of thing is quite hard to
keep track of... i.e. we don't know what is happening until we reboot,
etc....
grrr
Heck, Thomas, I certainly wasn't taking it badly. If the issue is new
enough that I can't find anything on Google, then it's off to the
developers I go; no one else is going to know anything.
Meanwhile, I've got to juggle the flight school appointment here with my
trip to the Polynesian Culture Center (http://www.polynesia.com); I'll
probably talk to you all next when I'm back in St. Augustine. Cheers!
[Kapil]
There is a nice "comparison table" by Nigel Cunningham regarding the
swsusp patches available at swsusp.sf.net.
Basically, there are three ways to get suspend-to-disk working for 2.6
kernels. There are two methods that are in the default kernel tarball.
There is also a "swsusp2" patch that can be downloaded from the
swsusp.sf.net site.
What is not mentioned (and not configured into the Kconfig scripts) is
that the current versions appear to depend on the "module unloading"
fature that is optional with the 2.6 kernel.
Secondly, while it is true that
echo -n disk > /sys/power/state
or
echo 4 > /proc/acpi/sleep
or
echo 1 > /proc/kernel/suspend/activate or whatever
should suspend-to-disk, it seems to be much smoother to use the "hibernate"
script that can be downloaded as part of the tarball at swsusp.sf.net.
Finally, my own experience with the 1.0.3 patch for 2.4 is that it works
quite well. I haven't had as much stability with the 2.6 suspend
versions. (Works only while testing but fails when actually required
.
Eventually, I'll get around to posting my questions to the swsusp site
once I have all the variables sorted out.
[Heather] Hey gentle readers! If you know more on this topic please feel free to
drop us a note - inquiring minds want working laptops
Oh-oh. This isn't looking good for the moment...
From Ben Okopnik
Answered By: Robos, Ben, Karl-Heinz Herrman, Jason Creighton, Thomas Adam
X11 and acer laptops
~~~~~~~~~~~~~~~~~~~~
This is by far the longest thread of the month with over fifty replies
and many that I have deferred. I have tried to split it into sections
where appropriate.... enjoy!
-- Thomas Adam
I've got this brand-new Acer Aspire 2003LMi laptop - slick, sleek, and
faster than a greased weasel on dexedrine. However... it doesn't seem to
be too Linux-friendly.
(( Sorta. Here are the details:
I can boot LNX-BBC, and even run X. So far, so good - but this is a
really high-res display with lots of fancy goodies. Besides, although
I remember LNX-BBC as having some sort of an HD install procedure, it's
really not what I want to install here.
I can boot Knoppix. Well, more or less, with a lot of emphasis on
"less". It comes up, gets through the KDE stuff... and freezes about 10
[Robos]
Try disabling acpi. In my laptop, if I move the mouse and kde wants to look
what the battery status is the mouse jumps all over the place. Maybe yours
is worse.
append acpi=off should do the trick.
[Ben]
Interestingly enough, ACPI is what works here, while APM doesn't. I've
had to install the entire ACPI kit'n'kaboodle (kernel recompilation,
userland stuff) and remove APM to make it work. Don't have suspend going
yet, but all the other power management stuff works.
[Thomas]
ACPI overrides APM in its functions and callings. On my laptop it works
rather well, and coupled with swsusp is a dream to suspend to disk.
To get swsusp going, I cannot do better than to recommend the patch for it
from:
A kernel compilation is again going to have to ensue, but afterwards, you
can do cool things like:
sudo echo 4 > /proc/acpi/sleep
which will dump everything onto your swapspace.
That does look pretty cool... however, I'm not really all that wild
about having to maintain a stable of patches. I might do it anyway - I
really want that "suspend to swap" feature - but it just seems like an
annoying thing to have to remember in addition to the kernel upgrades.
seconds after I start moving the mouse. "knoppix 2" (console only) works
fine. "fbdev", "vesa", etc... none of the servers that I've tried for X
work in any reasonable way.
[Robos]
What's the graphics-card?
Radeon Mobility 9200 (9M+). New enough that "pci.ids" in the 2.6 kernel
(I had to upgrade - just moving the mouse (Synaptics touchpad) would
completely lock the machine with 2.4.22!) doesn't have it yet, although
the folks that actually maintain "pci.ids" already have it. I'd tried,
several times, to use the "experimental" xfree86 server, which
supposedly can handle it via the "ati" driver, but no luck - so I'm
stuck with the proprietary ATI driver. However, I'm a lot happier with
that than Dell's nVidia; at least I know that in a year at most, ATI
will release the interface.
[Thomas]
Yeah, the experimental xserver-xfree86 package is the one you're going to
have to use for this. I apt-get'ed it the other day.
There are four partitions on the new machine's HD: Wind0ws C:, Wind0ws
D:, "suspend", and... ready for this?... Linux. I kid you not. This
gadget has what's called "EFI", which I just happened to read about the
day before I bought it by pure coincidence (I had no idea that this
machine would have it) - it's a BIOS "replacement" that allows much
easier pre-boot programming. What these folks have done, clever buggers
that they are, is create an app - two of them, really - called "Arcade"
(plays VCDs/DVDs, shows your picture repositories, etc. - sound
familiar? Think "MoviX", etc.) and put a button on the front of the
laptop. When the 'top is _off,_ pushing this button fires up "Arcade"
without starting Wind0ws. When it's on, it fires up a Wind0ws app
that looks exactly the same. Is this cool, or what? The way I found out
[Robos]
This sounds cool. I heard about it but quickly forgot again.
that it was Linux is by running LNX-BBC and looking at the partitions.
When I saw "/bin", "/boot", "/etc" and so on, I thought I was having a
brain cramp... took me a bit to figure it out.
Aside from that, though - obviously, FIPS won't work with this rig. What
I need to do is shrink that first partition (Wind0ws) and put Linux into
the newly-created second partition (I don't want to just blow away
Wind0ws - there are a few gadgets here, such as a built-in
SD/SmartMedia/etc. card reader that are going to take some research
[Robos]
I'd say they connected this to the internal usb plug and so this should be
easy to get working: enable "probe all LUNs" in the scsi section and then it
should show up with scsi-emulation on.
This is at least how my 6in1 card reader shows up when I plug it in (is
external)
[Ben]
before I can use them under Linux.) I wouldn't mind reinstalling Wind0ws
after tweaking partitions - but all I have are "recovery" CDs (which
will, presumably, restore the system to exactly the way it is now.)
[Robos]
partimaged? Have some place to put the image?
And I have had the experience with my backup-partition that when I did run
it (when you activate the playing-back mode in the bios) it started dos and
ran the win installer - LOL. But this way I was able to keep my partition
the way they were and have a c: which is still fat32.
Hmmm... I just thought of something. I could blow it all away, install
Linux - I'd have at least console-mode stuff, which would be sucky but
workable, and copy everything over from the old machine. Then, when I
come back, I could buy an external HD for backup, copy it all off,
reinstall Wind0ws, and be abck to the same status. Hm. Frankly, this
sounds really last-ditch. Worst of all, the new 'top has no serial
port, so I wouldn't be able to use my cell phone to connect to the Net.
[Robos]
Does your cell-phone have irda? Believe me, this is rather easy (if your
acer has irda too and this is supported). I was surfing in my holidays back
in the spring with my siemens ME45 lying beside my thinkpad r31. Connectivity!
[Ben]
I did too. Didn't work for me.
[Thomas]
The first thing I could be inclined you did, Ben, is provide your "lspci"
output, along with "lspci -n". What you should probably ensure is that
your PCI entry is valid for the detected card.
where nn is your numbers from the correct entry for your card, but they
HAVE to be in decimal format. lspci gives you them in hexadecimal and do
you have to convert them. Using my example then...
PCI:00:02:00
I want you to try VESA this time, Ben. See if that makes a difference.
However without knowing much more information about your card specifically
there is not much more I can do
You should ensure that you have AGP
support compiled into your kernel at least.
The thread then changes slightly, but we're still on similar lines
-- Thomas Adam
OK, so here's the follow-up: since Thomas prodded me, and I have a
weakness for unrestrained bouts of geekery anyway, I gave it one more
try... and got some good results (didn't have much to do with
configuring X, though. Sorry, pal.) I even got my framebuffer stuff
working, including Tux at boot time - yay!
Oh well, didn't hurt
-- Thomas Adam
I installed the 2.6.0-test11 kernel so I could do the "swsusp" patch -
haven't yet, the network here at the hotel died for an hour or so and
just came back up - and compiled it (I think this is where the
framebuffer stuff came from; I hadn't changed anything in that section.)
Previously, though, I got a very thin lead from a discussion I saw where
somebody recommended turning off AGP in the kernel config, since the ATI
driver does its own, followed by someone else recommending trying it
both ways - according to them, some ATI cards will only work with
ATI's AGP while some others work only with in-kernel AGP. Actually,
that didn't help anything, but while twiddling this, and making
corresponding tweaks in the ATI "fglrx" driver configuration, I got an
error from trying to load "fglrx" - 'The fglrx module must be loaded
before any other DRM driver!' From there, it was a short trip back to
the kernel, compiling the ATI Radeon DRM as a _module,_ then loading
"fglrx" followed by "radeon" in /etc/modules, and - /voila/! Well,
partially.
I now have 1280x800 video with 24 bit color (still not fantastic as
compared to what the card/screen can do, but certainly better.)
According to the messages in the X log, this is the best that the ATI
driver can do - which is pretty sad, but what can you do. Hopefully,
they'll make it better as time goes on.
"swsusp" and the card reader are next on the list...
(For those of you who think I'm blowing my time in Hawaii on this stuff,
wrong. I'm getting up at 4:30 a.m. local time - which is 9:30 a.m. by my
internal clock, which I'm lucky enough to have decent control over - and
banging away on this until it's time to go to work. Ooops - it's that
time now!)
If you try doing this from 2.4.X -> 2.6.X it will not work. In any case, I
always like to build kernels using a fresh config file no matter what I
last used.
[Ben]
Not at all. My first experience with the 2.6 kernel was this past week,
and I was greatly tickled by the fact that it automatically takes the
previous .config and (obviously) does a bit of intelligent parsing. You
can even (and this is what I did) do
make oldconfig
after installing the new kernel, and it will load up the old config file
and only ask you about the differences (there weren't any between 2.6.0
and 2.6.0-test11.)
Well, at this point, I've managed to take a largish step back. :/
"swsusp" docs say that you should have at least $MEMSIZE * 1.3 in swap,
so I blew off the empty partition that I had following swap, deleted
both of them, resized swap and created a partition out of what
remained... and now, when I boot, "fsck" falls down with a loud "THUD"
and cries that it found an error and I should re-run it as root without
"-a" or "-p". When I do so, it tells me I have a mismatch between what
the BIOS and the partition info say (about 2,000 sectors) and requires
me to press 'Enter' 6,000 times (it generates 3 error messages/prompts per
sector.) No, you cannot pipe the output of 'yes' to it. Laying a weight
on the 'Enter' key (hi-tech solutions is us!) gets it done in about five
minutes... after which it proceeds to repeat exactly what it did before.
However, in this case, "Hit Ctrl-D to continue normal startup" actually
does work, so here I am. [sigh]
[Thomas]
Hmm, did you use cfdisk for this? It might be that you have to have a new
map file in your /boot partition. Try running lilo again to see if that'll
kick it? It might be worth just disabling swap for the time being and
manually running:
swapon /dev/xxx
later on.
I used cfdisk. Deleted them both, made two new ones using the space. However,
it's fixed already.
Deleted them again, rebooted with them still as
free space (that way I knew that "fsck" couldn't complain about
partition mismatches there), then recreated them on the next boot.
Whew.
[Robos]
There have been sone synaptic touchpad fixes gone into 2.6.1 according to
changelog.
[Ben]
Not enough of them - tapping the touchpad, which has always worked, got
borked in the process, and "gpm" doesn't work with the "synaptic" driver
in the kernel (TONS of spurious messages.) Fortunately, I found patches
for both of them. Tapping still doesn't work in the console, but it's
not a huge issue. Also, the touchpad buttons are arranged like this:
where 'X' is the four-directional 'scroll' button... which makes it damn
near impossible to click 1+2 to paste. There's yet another patch that
allows 'up' on the scroll button to be used as button 3, but I haven't
managed to make it work yet.
Other than that, I'm actually doing reasonably OK with it for the amount
of time I've spent tweaking it:
Working:
1024x768 video
Sound
Touchpad (except the above issues)
NIC
USB2.0
ACPI (no suspend yet, though)
Untested but everything loads OK:
Wireless networking
Bluetooth interface
CD-RW/DVD-RW
Firewire
Parallel port
Video out
FIR
Not working yet:
4-in-1 card reader
Better video
Framebuffer
Modem (Lucent winmodem, bleh :( There's probably a binary driver
somewhere, but I think I'll get a PCMCIA modem.)
Five-way "media control" keys
[K.-H]
Dell Inspirons (and maybe others) had 1600x1200/15" for some years now
at least starting with the Insp. 8000 on which I'm typing. There were
Insp. 5000 with high res 15" screens as well. The current 8600 or higher
have an even higher resolution on the top models. As you sit rather
close to a lapscreen I like the display. I had to switch some fonts to a
slightly larger one, but the displays are very sharp at the native
resolution and a 10pt full A4 page in gv (antialias on) is quite
readable.
Currently I'm thinking about a desk TFT -- and am rather unimpressed as
you almost can't get more then 1024x768 up to 17", then 19" have
1280xWhatever. Admittedly that screen is further back from the keyboard,
but why is no company offering any TFT with a higher res. They are
possible after all.
The Inspirons use either nvidia Geforce 2 (or up in later models) or ATI
Cards -- which won't help you Ben. I hope you figure out how to get into
native resolution. For most others XFree4 seems to calculate good
modelines by itself (I've not specified any). As yours is slightly weird
maybe you need to specify one, keep looking on google.
Then there's the BIOS to look at...
-- Heather Stern
There are four partitions on the new machine's HD: Wind0ws C:, Wind0ws
D:, "suspend", and... ready for this?... Linux. I kid you not. This
gadget has what's called "EFI", which I just happened to read about the
day before I bought it by pure coincidence (I had no idea that this
machine would have it) - it's a BIOS "replacement" that allows much
easier pre-boot programming. What these folks have done, clever buggers
that they are, is create an app - two of them, really - called "Arcade"
(plays VCDs/DVDs, shows your picture repositories, etc. - sound
familiar? Think "MoviX", etc.) and put a button on the front of the
laptop. When the 'top is _off,_ pushing this button fires up "Arcade"
without starting Wind0ws. When it's on, it fires up a Wind0ws app
that looks exactly the same. Is this cool, or what? The way I found out
that it was Linux is by running LNX-BBC and looking at the partitions.
When I saw "/bin", "/boot", "/etc" and so on, I thought I was having a
brain cramp... took me a bit to figure it out.
[Jason]
Okay, so there is actually some version of Linux installed on the
laptop? That is really interesting. If you have time (Ha ha!), I'd
really like it if you could post some details sometime.
I don't have the HD installed (although I have it with me), but what I
remember of it is this: it's a minimal (sorta like a "chroot" jail)
system with a large tarball in "/". Looking inside the tarball revealed
a more complete system; presumably, it works something like the LNX-BBC
system (Oy, Heather!) by using the basic system to fire up, then
decompress the tarball into memory and "pivot_root" (I'm guessing here.)
[Jason]
BTW, when I first heard about EFI, my first thought was "Cool!". My
second thought was "Somebody is going to code up a Tetris clone for this
thing". Really, what could be more fun than playing tetris without
having to boot up an OS? Actually, I expect people to more or less treat
it like another OS, because, AFAIK, that's basically what it is. So
expect all manner of useless programs.
I've explored it a little further, and it seems to be some
Chinese-assembled version of Linux; fires up, opens up a tar archive,
chroots into it, and runs a rather flashy Linux app (a front-end for lots
of neat-o music/DVD/CD/etc. utils - mostly based on "mplayer", AFAICT.)
[Robos]
Well, you should then write the mplayer folks a nice email telling them
that. Quite recently they even started a new ml called mplayer-legal...
Or do you think you can get the sources somewhere?
[Ben]
It's quite cute, with a bunch of "lilo.conf.X" entries in /etc - obviously
in case the partition ends up being anything other than hda4 which is where
it is by default. Cuter yet, there's no alternate VT you can flip to - but
there is a getty running on a serial port, and "/etc/passwd" doesn't have
an entry for "root" but does contain one for "ava:0:0:..." Ain't that
preeeecious?
Heck, I might just copy the thing and put in on a CD for a stand-alone
movie/slideshow/etc. player.
I've actually considered doing that [a howto], on the principle that if I had that
much trouble, other people surely will as well. However, I've held off so
far because I'm not quite finished yet; I've zapped the original HD back
into the Acer, and am now trying to split hda1 into hda1 and hda5, for
Wind*ws on 1 (10GB or so) and Linux on 5 (40GB). However, the DVD+R media
that I bought for backing up hda1 turns out to be incompatible with the drive
(which does only DVD-R and down), so I have to go back to the store.... all
this in between yoga classes, a flight medical (I now have my class III
flying license), actual flying, chanty singing with a reenactment group,
trying to somehow get Net access going (I've sorta got it - by buying a new
cellphone), patching a leak that my dinghy has mysteriously developed,
dealing with a couple of clients who have picked this time to play some
highly shitful games with paying me (until I threatened legal action),
fixing a recalcitrant propane heater in my aft cabin, and handling the
initial stages of what looks like a local romantic involvement.
Actually, Ben did just that. The URL for which is here:
"There are two ways of constructing a software design. One
way is to make it so simple that there are obviously no deficiencies,
and the other is to make it so complicated that there are no obvious
deficiencies." - C.A.R. Hoare
"Sure I'm paranoid, but am I paranoid
ENOUGH?" - Unknown
Introduction
System administrators frequently want to be able to work on the machines
they run even when they are far away from them. There are secure tools
that allow full remote shell access, like ssh and lsh, but due to their
complexity they have suffered critical exploits from time to time.
In addition, their overhead can be excessive for some purposes. Fortunately,
other options are available that can be used alone or can be combined with
remote shells to create a more secure overall system.
Overview
Maybe the pager has just gone off when you're home in bed, and the boss
wants you to fix the broken database now. Or perhaps you're out
for lunch and someone calls to tell you the mailserver has been cracked
and is currently spamming the world, and you need to bring it down fast.
Possibly you've checked and your Web server has wedged itself and needs to
be restarted. Or suppose you're just on vacation and find you want to
update your home Web site with some new photos. In all these cases, you'd
like to do something to the machine over the Internet without having to
actually sit in front of it - things you don't want just anybody
to be able to do.
The Problem
Tools like ssh and lsh are great for allowing
secure remote access to your system. They offer essentially full, flexible
remote control of a machine, in an encrypted and authenticated manner. But
they are complex pieces of software; there's no way to do what they do
without being complex. And with complexity comes bugs. SSH and lsh,
and related tools like Webmin, have
all had serious flaws that would allow an attacker to get full control over
your system. Leaving them available all the time is a risk - sometimes it's
necessary, but it's still a risk. And in some cases, you'd like to be able
to tell the machine to do something, but it's not even attached to the
network on a regular basis.
Some Solutions
It would be nice to enable remote shell access only when necessary. And
perhaps (for something like shutting down a mail server) you don't even
need a full shell, just a way to fire off a script remotely. Of course, the
problem then becomes, how do you know that the alternative software is any
more secure than ssh itself? Various people have worked on this problem in
the past, and several potential solutions are available, ranging from the
simple and venerable to the new and exotic.
Xringd uses a modem to control a machine remotely.
Mail filters can be used to trigger actions based on
special messages. Some solutions (like 'port knocking'
and 'Net::Pcap') use the network, but without requiring
even a single open port. Lando runs commands over a
network, using username and password. Most recently, a program specifically
for secure remote execution called Ostiary has been
developed.
The eXtended Ring
Daemon, or "Xringd", uses a modem to monitor rings on a phone line. It
counts the number of rings, and the time between them. If a 'sequence'
matches one of the ones that it has been set up to detect, Xringd will run
an associated command.
This is very nice from a security perspective. Since it uses no network
connection at all, it's entirely immune to network attacks like buffer
overflows. It can be used even when a network connection is unavailable
(it's often used to cause a computer to initiate a dialup connection).
The only 'client' you need is a phone. If you use it to start up ssh on
demand, then the attacker needs to know the right phone number and the
right ring pattern - it's quite hard to sniff that kind of thing remotely.
It's also highly resistant to a man in the
middle attack. (If you have to worry about someone rerouting your
phone calls, you're in more trouble than Xringd can save you from.)
There are some practical issues that may make this unattractive in
some circumstances. You need a modem and a telephone line to the server.
(Fortunately, you don't need a fast modem at all; even a 1200 baud
one will do nicely, but some servers are not placed close to a telephone
jack.) Also, things like answering machines or voicemail (or even other
people answering the telephone) can interfere with Xringd. If you give the
server a dedicated line, you can avoid these problems, but that can be
costly.
Finally, note that the rings you hear when making a call are not
necessarily synchronized with the ring signals actually sent to the
telephone. In most circumstances, they are close enough, but reliability can
be an issue at times.
Most of the mail filtering programs have a way to invoke scripts when
mail matching a pattern is received (in the simplest case, mail to a
particular address). Assuming the server is running an SMTP daemon,
this can be a nice way of triggering
actions remotely. Technically, one could even send a shell script
to be run, and have it e-mail the results back to you, giving you the
equivalent of a very slow remote shell. The only client needed
is an e-mail program, or even a webmail account.
The first problem is that if the box you want to talk to doesn't accept
e-mail, this obviously won't work. (Adding an entire mail server, with the
attendant risks of bugs, spam load, etc., just for remote execution doesn't
make a lot of sense.) Some machines only periodically collect e-mail from
a primary server, so there can be a substantial delay between when a
command is sent and when it is acted upon.
Furthermore, if you don't encrypt the traffic in some way (or at least
sign it with PGP), then anyone sniffing traffic between you and your server
may be able to take advantage of the same channel to do mischief, or
perform a man-in-the-middle-attack. (E-mail traffic is notoriously easy to
falsify; hence the avalanche of spam these days.)
CVTSA, or "ClairVoyanT
SysAdmin", is a system designed specifically for running commands
through e-mail. It has some support for using passwords, but does not
(currently) encrypt them in transit, so a sniffer could capture them
and use them again.
Of course, if the only things you want to do with this type of system
are emergency shutdowns and other such (hopefully rare) crisis management,
then even an unencrypted channel might work. However, you'll need to
change the 'magic trigger pattern' each time after you use it, or you
take the risk that an attacker might capture it and 'replay' it at an
inconvenient time.
With port
knocking, a daemon monitors firewall logs, looking for particular
sequences of connection attempts to particular (closed) ports. When it
sees a sequence it recognizes, it runs the associated command. This
isn't terribly bandwidth efficient, but it has some nice properties.
First, it's hard to tell if a server is listening for port knocks.
Second (and most important), it's awfully hard to crack a closed
port. (Linksys routers have had a simple version of this for a while,
BTW, that they call port
triggering.)
However, a clever attacker with a sniffer could notice this traffic,
and duplicate it for their own use. More complicated encodings could
express something like a PGP signature (indeed, in theory one could
create an entire network protocol based on port knocks), but things
rapidly become difficult to work with. As with 'mail filtering'
solutions, one can either use it sparingly in emergencies, or move to
real cryptography.
It's also important to realize that this system is critically dependent
on the probe packets actually being delivered, and delivered in the
order that they were sent. This is not guaranteed on the Internet.
What's more, depending on where you're at (e.g., an Internet cafe or
behind a business firewall), you might not be allowed to connect out to
arbitrary ports. The more complex you make the 'knocks', the less
reliable the system will be.
Also, notice that at least one entire IP packet (28 bytes or so minimum)
is used to transmit roughly one bit of information. In terms of network
efficiency, it's almost hideous. For a simple 'open up ssh' message, it's
not a consideration, but actually adding cryptographic security to this
system could use up a decent chunk of the available bandwidth.
Finally, this increases the CPU load for each entry in the firewall
log. Depending on how detailed the logs are, and how fast and busy
the network is, this can be a significant drain on resources.
Another interesting approach is to use Net::Pcap
or other network capturing software to look for specific packets on the
network (e.g., DNS requests) and examine them for particular data (e.g.,
a particular address). If found, it can enable ssh temporarily, or
perform other actions.
One potential benefit of this approach is that a computer doesn't have
to have an address on a network in order to monitor traffic on
that network. You can set the card to 'promiscuous mode' and examine
all the traffic on the wire. (It's very hard to hack a machine
you don't even know is there.) Once the 'trigger' is spotted, the
sniffer can use other means (a separate network, a serial link, even
Xringd) to open up SSH on a target machine. Of course, you can also
simply run the sniffer directly on the target.
Again, a clever attacker with their own sniffer may be able to detect
the unusual activity and correlate it. To make this system truly
secure, you would need more complex encoding/encryption of the 'trigger'
traffic.
Additionally, the CPU load for this solution can be even worse than for
'port knocking' systems. A 'port knocking' daemon monitors firewall
logs, which can have variable levels of detail. By necessity, a
'sniffer' solution must examine every packet on the network
segment, which can be a substantial task for a busy gigabit line.
Lando allows a user to
run a preconfigured set of commands remotely, using passwords, and even
allowing the user to supply arguments to them. While it currently has
only a Windows client, and passwords are sent in the clear (making it
suitable only for use on a trusted local network, or perhaps on a VPN),
it can be very useful for, e.g. operating a local firewall box without
going to the trouble of logging in.
All of the above solutions have their advantages, but each has some
practical issues that can make them unsuitable for particular
applications. Ostiary was designed
to be a secure alternative that uses minimal resources. It tackles this
problem with what might be termed "aggressive simplicity". It does
require an active connection to the network (unlike Xringd and sniffing),
but allows for much better default security with very low CPU, RAM, disk,
and network bandwidth requirements.
An Ostiary server has one open port that it listens on. When someone
connects, the server sends a random fixed length 'salt' message 16
bytes in size - the size of an MD5 hash. It
then waits (with a timeout) for a reply from the client. It reads (at
most) 16 bytes of reply, and closes the connection.
Ostiary has a list of commands to run, with associated passwords. It
runs through the list, and hashes these passwords with
the 'salt' it sent to the client. If one of these hashes matches the
reply from the client, the associated command is run. (One final touch
is that a record is kept of connections, and clients with too many
failed attempts are 'locked out', and all subsequent communication from
them is ignored.)
A detailed security
analysis is available, but a few things about this system should
be clear. With a protocol this simple, the chances for dangerous
bugs are drastically reduced. Using fixed-length messages essentially
eliminates the chances of a buffer
overflow or other memory error. (Indeed, Ostiary does no dynamic
memory allocation of any kind - everything is stored in static,
fixed-size data structures.) Replay and man-in-the-middle attacks are
also effectively useless. Ostiary limits how fast it accepts connections,
enforcing low CPU and network usage. (The first production Ostiary server
was a 16MHz 68030 machine.) Client requirements are even lower: Clients
are available for Palm Pilots and even Windows.
Unlike a procmail-based solution, where you can put arbitrary commands (with
arguments) in the message, Ostiary can only run the fixed set of commands
you have preconfigured. The only argument it supplies to the commands is the
IP address of the client that initiated the command. It requires an active
network connection (unlike Xringd) and an open port (unlike port knocking or
sniffing), which may entail configuring a firewall to open a new port.
(Although one could run Ostiary on, say, port 22, and upon receipt of
the correct command, it could terminate itself and spawn sshd...)
Since Ostiary uses TCP, it is as reliable as the network it uses to
communicate. Problems with miscounted phone rings (a la Xringd) or randomly
dropped packets (a la port knocking) are not a concern.
Summary
The following table summarizes the pros and cons of the various systems
outlined above. "Replay" and "Man-in-the-middle" indicate if the default
system is vulnerable to the corresponding attacks. "Command arguments"
indicates if the system can run arbitrary commands with arguments. "CPU
load" indicates that CPU time can be a significant consideration. "Special
client" indicates that a specific client program is needed to work with
that system.
System
Xringd
Mail filter
Port knocking
Sniffers
Lando
Ostiary
Network Required?
Yes
Yes
Yes
Yes
Yes
Port Required?
Yes
Yes
Yes
Modem Required?
Yes
Replay?
Yes
Yes
Yes
Yes
Man-in-the-middle?
Yes
Yes
Yes
Yes
Command arguments?
Yes
Yes
CPU load?
Sometimes
Yes
Yes
Special client?
Sometimes
Sometimes
Yes
Yes
None of these approaches is right for everyone. But all of them can be
used to make attacks at least more inconvenient, and in many cases far
more difficult. Remember, though, to analyze their pros and cons
relative to your specific situation. Also remember that true security
is a process, not a goal - you can never just install some software
and be done thinking about it.
Ray Ingles has been involved with Linux since 1995. In
addition to being an active member of the
Metro Detroit Linux User's Group,
he has made minor contributions to the UPS HOWTO and the Linux
Joystick Driver.
Where do you go to find out what's on TV? The usual suspects might include
a newspaper, a recent issue of TV Guide magazine, a favorite Web site, or your nearest
TiVo, ReplayTV, or other PVR. But don't forget to add Linux to the top of that
list. You can let the machine do the dirty work and bring the listings to you. XMLTV, a
short bash script, and a cron job are all you need to get started.
Installation
First things first: getting the program installed. XMLTV is a suite of Perl
scripts and can be downloaded from
membled.com/work/apps/xmltv.
There are releases for Unix-like and Windows environments, but, for obvious reasons,
we'll focus on the former. If you're installing from source, it's the usual
routine:
% perl Makefile.PL
% make
% make test
% make install
If you're on Debian, it's all just an apt-get away
(apt-cache search xmltv). Links to packages for OS X, Red Hat 8, and Red Hat 9 are
available from the project's homepage.
Configuration
Before XMLTV can be useful, it needs to know where in
the world you are. XMLTV is international it can fetch TV listings for
Canada and the United States, the United Kingdom, Austria and Germany, New
Zealand, Finland, Italy, Spain, the Netherlands, Denmark, and Hungary. (Belgium
and France are in the works.) The scripts that collect listings for a
particular country are referred to as grabbers, and you'll find them on the
command line under tv_grab_*. We'll use the U.S. grabber,
tv_grab_na.
When you first run the grabber, do so with the --configure option. This
starts a question-and-answer session in which you and the grabber get a little
bit better acquainted, as far as your Zip code, TV service provider, and
channels you want to ignore are concerned. The results of the script are written to
~/.xmltv/tv_grab_na.conf, and can be easily edited by hand.
At this point, XMLTV is ready to do your bidding. Do a man tv_grab_na to learn
about all the available options. For now, just two will suffice:
% tv_grab_na --days 1 --output /tmp/tv.xml
This tells the grabber to get one day's worth of listings, and save them out to /tmp/tv.xml.
XMLTV's file format doesn't quite make for friendly reading, unless you
enjoy reading raw markup. A few more scripts from the suite can fix that. tv_sort
sorts the contents of an xmltv file date.
tv_grep lets you weed out some of the obvious cruft in the
listings. Here's how I run it:
The commands above sort the original file and then discard anything
categorized as "Children" or "Sports", and anything with "Paid Programming"
(infomercials) or "Local Origination" (public access) in the title. Also,
we're discarding everything that aired before the script ran.
At this point, we've still got an XML file. Converters to the rescue!
tv_to_text is one of the tools that can help us go from XML to something else.
(Other possibilities include LaTeX, HTML and PDF. Check the readme to see what's
currently available.) After running something like this:
21:00--21:30 Spy School 38
21:00--21:30 Designing for the Sexes // European Kitchen 64
21:00--21:30 Chappelle's Show 67
21:00--21:30 The Real World // Las Vegas 71
21:00--22:00 Law & Order: Special Victims Unit // Guilt 44
21:00--22:00 Wild Card // Auntie Venom 45
21:00--22:00 Cold Case Files // The Accidental Killer; Little Sister Lost 57
21:00--22:00 America's Most Wanted: America Fights Back // Top Ten Most Wanted Fugitives 5
21:00--22:00 The FBI Files // The Price of Greed 60
21:00--22:00 Trading Spaces // Nashville: Murphywood Crossing 61
21:00--22:00 Great Chowder Cook-Off 63
21:00--22:00 Ends of the Earth // Secrets of the Holy Land 65
21:00--22:00 The E! True Hollywood Story // The Hilton Sisters 68
...
Simple and no frills. Just what we need for the final step: e-mail delivery.
Delivery
If we stopped at this point we'd have used several of XMLTV's abilities but
hardly anything else. We'd also be running low on convenience and automation.
Fortunately, we can wrap all the commands we've seen so far into a shell
script, and have it e-mail us the final results. mail can take care
of, well, the mailing:
% mail -s "Today's TV listings from XMLTV" user@localhost < /tmp/tv.txt
#!/bin/sh
# Grab today's listings:
tv_grab_na --days 1 --output /tmp/tv.xml
# Sort
tv_sort --output /tmp/tv_sorted.xml /tmp/tv.xml
# Grep
tv_grep --output /tmp/tv_grepped.xml --ignore-case --not --category Children \
--not --category Sports --not --title "Paid Programming" \
--not --title "Local Origination" \
--on-after now /tmp/tv_sorted.xml
# Convert To Text
tv_to_text --output /tmp/tv.txt /tmp/tv_grepped.xml
# Email
mail -s "Today's TV listings from XMLTV" user@localhost < /tmp/tv.txt
Put that in a cron job that runs once per day, and you've got TV listings with no outside advertising,
and no channels or shows you know you aren't interested in.
More importantly, you've got a foundation to build on. What we've
covered is just the beginning beyond the command-line scripts, a
GUI client is also available. Of course, there are plenty more things you could do from the
command line, such as:
Pull in data from imdb.com via tv_imdb
Split the listings into separate files for each day and channel via tv_split
Transform the XML with your own XSLT stylesheet.
Only send e-mail if certain keywords are found
It all depends on how you want to consume the information, and how cleverly you can chain all the scripts
together.
Bill Lovett is a Web developer in New York City. He's one of those PHP/MySQL
types. And he has this weird thing about running Linux on old machines that by
all rights should have been trashed years ago. Read more about Bill and his
Open Source projects at www.ilovett.com
Many of us make a living out of Linux - but, if
somebody asks us why we are so crazy about it, one
common answer would be `fun'. Playing with Linux is lots
of fun - with the added benefit that, most of the time,
you end up learning a lot. Recently, I happened to come
across a nice book which tries to emphasize the `fun' aspect
of Linux - it describes several small `projects' (a jukebox,
a picture frame, etc.) that a moderately experienced Linux
user may be able to implement on her own. One of the projects involved
interfacing with a temperature-sensing element and putting up
the temperature value on a Web page (or including it in your email
signature - and any other crazy stuff which you can imagine!). The
only trouble was that, in the part of the world where
I live, walking up to an electronics store and asking for an
integrated, 1-wire temperature-sensing element is
more likely to yield a hard stare than anything else. Smart sensors that
can be directly interfaced to the PC with the minimum of fuss
are seldom available off-the-shelf - you will mostly have to `roll your own'
- which adds to the fun and excitement. With a low-cost general purpose
microcontroller like the PIC16F628, bits and pieces of cheap, commonly available
electronics components, and LOTS of code, you can build many interesting `toys'
and hook them up to your Linux machine - a really great learning experience
for the hardware hacker who wants to learn Linux, or the Linux hacker who wants
to learn a bit of hardware. This article describes how I went about building
my temperature-sensing project - amateur Linux/hardware hackers might find some
of the ideas useful when they start building things on their own.
Get a PIC micro, and set it up to work with Linux
This is the first step. Microchip PIC controllers are commonly
available. If you are like me, working with a soldering iron
for more than 10 minutes would drive you crazy - so you have to
choose the right kind of PIC - the one that can be programmed
with the simplest possible circuit (connected to the PC parallel
port), preferably with a 5V supply. Look no further than the
PIC16F628. This is a cool device that has lots of peripherals (except
the ADC - but then, we can roll our own crude analog-to-digital
converter with the comparator and pulse width modulation facilities
offered by the PIC) and supports a `Low-Voltage Programming Mode'. I
found a nice little circuit (the simplest circuit, and one
that works perfectly, out of the dozens I have seen on the Net)
designed by Jim Paris for a microcontroller programming laboratory
at MIT. Here is the circuit:
I assembled the circuit on a breadboard for testing in a
few minutes' time.
Jim Paris has designed a program (called `jimpic') for
burning machine code onto the flash memory of the microcontroller. It is
available for download from here. I
wrote a simple assembly language program, converted it into
machine code with the help of the `gpasm' assembler available
as part of the GNU PIC
Utilities Project and burned it onto the micro by running `jimpic'
with the `-b' option.
A Quick Introduction to PIC programming
A nice thing
about the PIC is that, if you have some background in general microprocessor
architecture and assembly language programming, you can become productive
with it in just about one or two hours' time. The instruction set is
very compact (35 instructions) and sufficient for most simple bit-twiddling
tasks. The PIC16F628 packs a decent 224 bytes of data memory with 2K of program
(code) memory. The peripherals include general-purpose digital I/O ports,
three timers, two analog comparators, on-chip voltage reference module,
Universal Synchronous-Asynchronous Receiver Transmitter (for serial communication),
and Capture-Compare-PWM module. Special CPU features include a watchdog timer,
brown-out detect circuitry, and an internal RC oscillator (so that you won't be
needing an external crystal if you aren't too concerned about precise timing).
The general purpose data RAM begins at address 0x20 (the locations below this
are Special Function Registers - basically memory mapped I/O ports,
control registers etc.). Here is an elementary assembly language program, which
simply stores the value 0 into the accumulator (the `W' register, in PIC
terminology).
(Remove the .txt extension if you download the listing. It's there only to
ensure browsers display it properly.)
We will now assemble the file:
gpasm -a inhx8m a.s
The result is an Intel hex format file, which can be given to `jimpic' for
burning. Each line of the hex file contains a few bytes of machine code,
the address at which the machine code is to be stored (in the flash memory
of the microcontroller), some kind of checksum, and some other information. Here
is the hex file generated by running `gpasm' over our assembly language program:
:020000000030CE
:02400E00983FD9
:00000001FF
The first line of our program tells the assembler that
machine code is to be generated for the PIC16F628. The second line includes
a file (available with the `gputils' distribution) that contains lots
of symbol definitions. The third line, a __CONFIG directive, tells the
assembler what special features of the microcontroller (say, the Watchdog
timer) should be enabled/disabled by writing bit patterns to a `magic'
`configuration word' within the PIC; _WDT_OFF means we don't want the watchdog
to be enabled, _INTRC_OSC_NOCLKOUT means we are going to use the internal
oscillator to provide the timing signals necessary for program execution.
You will have to refer to the 16F628 datasheet to know more about these
configuration bits. The fourth line is the only proper assembly language
instruction in the program - it moves the `literal' (constant) value 0
to the `W' register. Note that each line begins with a tab.
Lighting up an LED
Here is a program that lights up an LED connected to the RB0 pin of
the microcontroller:
PORTB is an eight-bit port - the direction of each pin (i.e., whether the
pin is to act as input or output) is controlled by individual bits of
the TRISB register - if a TRISB bit is set, the corresponding PORTB pin
is input - otherwise it is output. The PIC has the concept of `banked' addresses, which
is rather confusing to the beginner. (It's a headache even if you
are an `experienced' developer.) You visualize `banks' of special function
registers - the STATUS register is the same across all the banks while
the TRISB register is available only in bank 1. You are by default in bank 0.
To access TRISB, you have to `switch over' to bank 1. This is by setting the
RP0 bit of the status register. (When you read microcontroller manuals, you
will see that not only are the control registers given special names, even
the individual bits are named. Header files available with the development
kit for the microcontroller map these symbolic names to the numbers given in
the manual, making the life of the assembly programmer a bit easier.) The `bsf'
instruction (bit set f - `f' represents the fact that the number that comes
as the operand for the instruction represents a memory address or a special
function register and not a `literal') takes two operands - the first
one being the address of a RAM location or a special function register, and
the second, a bit number. The `movwf' instruction copies the contents of the
`W' register to the memory location whose address is the operand of
the instruction.
Building a `running' circuit
After assembling and burning the above program, we are ready
to see it in action. The running circuit can be built in a
jiffy - place +5V on the VDD pin of the PIC (pin 14), connect
Vss (pin 5) to circuit ground, connect MCLR (pin 4) to +5V through
a 2K resistor, connect the LED between RB0 and Gnd with a current
limiting resistor of say 1K in series - and that's all. You should
see the LED lighting up as soon as you apply power. Your next
attempt will be to make the LED blink - for that you will have
to read a little bit more about the PIC instruction set - the manual
will come in handy at this juncture.
Debugging tips
Here are some things that I have found handy while debugging:
Check the power supply
Don't jump into conclusions that the hardware is
wrong - you might have misinterpreted the datasheet, your
program logic might be wrong, or
worse still - the data sheet might be WRONG. Which brings
us to the next rule, which is:
Always read the manufacturers' errata - if they have
one. The 16F628 datasheet contains some errors
- especially concerning writes to EEPROM data memory and
the behavior of the MCLR pin in low-voltage programming
mode.
Don't think the hardware will never malfunction
- for example, the PIC might consume larger current when
writing to the internal data EEPROM; your battery-powered supply might
not be able to deliver the required current, and your program will
misbehave. If you have an external crystal, it might not be working
properly, and the micro might not be getting its clock.
Google Groups is your friend - use it wisely. Search the archive;
somebody might have experienced the same problem before. Post a message
if you feel that your problem is something `original'.
The temperature sensor interfacing project
The LM35 is a commonly available calibrated temperature sensor
that converts temperature (in degrees Celsius) to
voltage - each degree rise in temperature results in
10mV rise in output voltage. It's a three-pin device -
Vcc, Gnd, and voltage output. You can get the datasheet
from here. Say the current temperature
is 23 degrees Celsius; the voltage output would be 230 millivolts.
The question is, how do you convert this voltage to a
digital value? The easiest way would be to use a
commodity analog-to-digital converter,
and interface it to the printer port. Another solution
would be to use a PIC with a built in ADC (say the 12F675).
The third would be to use some of the peripherals available
in the 16F628, write some code, and build a crude ADC of
your own. As I had explored the first two options a lot in
the past, I thought of trying out the third one.
Two peripheral features of the PIC are of interest to
us here - one is the builtin PWM module (Pulse Width
Modulation), which is capable of generating, in, hardware,
a continuous stream of digital on-off pulses whose duty
cycle can be varied simply by storing certain numbers in
specific special function registers. Once the PWM module
is initialized to generate a pulse train of a specific duty
cycle, it will keep on doing so without any software intervention -
our program can do something else.
The PIC is also equipped with two analog comparators, which
can be configured in a variety of ways. Let's say we are using
just one of the comparators. Two PORTA pins can be programmed
to accept voltage levels and transmit them to the Vin+ and Vin-
pins of the comparator. The comparator output is high if the
Vin+ voltage is greater than the Vin- voltage, and low otherwise.
The output can be made available on another PORTA pin, or it
can be simply read from a particular bit of the Comparator Control
Register, CMCON.
Filtering PWM pulses
The figure shows a PWM pulse (off OV, on +5V) of period T being fed to an RC circuit (R*C >> T).
If the on-off periods are equal, the output seen across the capacitor
will be a constant DC level of magnitude 2.5V. Electrical engineering text books
should give you the reason why it is so - or, if you are not very sure of
the math involved (as I am), play with some R and C values until you get the
desired effect. Now what if you feed a PWM pulse whose on-time is less than
T/2? You will see that the output is again a DC level, but the magnitude has
come down proportionately. What if you increase the on-time? Again, the
output is a DC level, only thing is the magnitude has increased proportionately.
Now you have a cool way to implement a DAC, a digital-to-analog converter.
Say you want to generate a voltage of 0.449V. What if you program the PIC
so as to generate a PWM pulse train of period 256 microseconds and on-time
128micro. The output voltage would be 5V*(128/256.0) = 2.5V. Now, what if
the on-time is 23 micro seconds? The output is 5V*(23.0/256) = 0.449V. (I
use Python to do these quick-and-dirty calculations. It's one of my
favourite uses of this great language.) The
on-time can be altered easily by writing some numbers to two registers, CCPR1L
and CCP1CON. A pure digital way to generate analog voltage!
From DAC to ADC
What has all this got to do with converting the LM35 sensor's analog
voltage output to a numerical value? Well, a DAC, together with a comparator,
builds up an ADC. How? Say the maximum and minimum temperatures at your
place of residence can never go above/below 45 degree Celsius and
20 degree Celsius. So the sensor's output will always be between
.45V and .2V (remember, 10mV per degree change in temperature). We
start generating a PWM signal of period 256 microseconds. The RC-filtered
output is fed to Vin+ of the comparator, and the sensor's output is
fed to Vin-. Let's say the sensor output is .3V. If the PWM on-period
is 23 microseconds, the filtered DC level would be 5*(23.0/256) = 0.449V.
So, Vin+ is greater than Vin-, and the comparator output (as obtained
from a bit of the CMCON register) is high. Now, we start bringing down
the on-time. At a certain point, Vin- will go above Vin+, and the comparator
output drops to zero. The magnitude of the on-time at this point is
a true representation of the analog value of the sensor output. We communicate
this number to a program running on the Linux box through a serial
link. You can download the PIC assembly language program that does
all these tricks:
Instead of performing a `linear' search from the high boundary down to
the lower one, we can think of generating a voltage that lies in the
middle of this range and comparing it with the sensor output. If the
comparator says that the sensor output is higher, we can repeat the
same procedure on the upper half. This is the classical `binary search'
applied to solve a hardware problem! Horowitz and Hill, in their
book The Art of Electronics, have oscilloscope traces of this
binary search in action! Computer programmers should always show a good
amount of skepticism towards code that claims to do binary search -
the algorithm looks deceptively simple - but is in fact not very easy
to implement correctly.
Back to Linux
The PIC micro sends the temperature data it has gathered out through
a port pin (RB2) in a serial manner - this port pin is directly connected to the
receive pin of the PC serial port. What remains is to write a program
that will read this data and process it in some manner. Even though
the RS-232C serial communication standard defines an `on' voltage to be
between -3 and -12V and an `off' to be between +3 and +12, I have been
able to get satisfactory results using the 0 and 5V logic outputs from
the PIC port pin - if it doesn't work out for you, you will have to
place a device like the MAX232 between the PIC port pin and the
PC serial port receive pin.
Interfacing with the serial port
Let's look at the simplest way to interface an external
circuit to the serial port. (We won't be sending any data
out through the PC serial port - that would make the circuit
a wee bit more complex.) Pin number 2 of
the 9-pin PC serial port connector is the receive pin, 3 the transmit
pin, and 5, Ground. Let's say the PIC is sending data out
through its RB2 pin at 9600 bits per second, 8N1 (8 data bits,
no parity, 1 stop bit) format. The UART that controls the
PC serial port should be programmed for this particular baud
rate and data format. This can be done by writing magic bit
patterns to certain control registers. Once that is done, our
program can keep on polling a bit of the UART status register
to know whether a new data byte has arrived. Here is the code
listing:
The program has two disadvantages. One, it is using low-level
I/O calls, which, if they are to work properly, should be
preceded by an iopl() call. Only the superuser can call
iopl() successfully - so the program should run under root privilege.
We are wasting CPU time when we keep polling for data in
a tight loop; that's another big problem. Both are solved by
not directly interacting with the hardware - we can make use of
system calls to talk to the serial driver within the Linux kernel -
which will do all the low level stuff needed to manage blocking,
interrupt driven I/O.
Serial I/O in Python
The Python `termios' module can be used for doing serial comm
at a higher level. Isaac Barona Martinez has written
a simple wrapper over `termios'. It is called uspp and
is available for download from here. Using
this module, reading from the serial port is a breeze:
from uspp import *
# COM1 is initialized at 9600 baud. The
# default data format is 8N1
s = SerialPort("/dev/ttyS0", None, 9600)
s.flush() # discard unread bytes
print ord(s.read()) # s.read() returns a one-character
# string. We convert it into its ascii
# value
Once you get this far, let your imagination run riot!
Write a simple server that accepts connections over
the network and transmits the current temperature
Write a program that keeps on reading the temperature at
say, half-hour intervals. The temperature reading, together
with some stupid message like `Oh - it's burning hot here' can
be placed at the end of your `.signature' file!
Another idea would be to use something like the Python
`ftplib' to upload the temperature reading to your Web server
periodically.
Acknowledgements
Thanks to Christopher Negus and Chuck Wolber
for a really cool book!
Thanks to Jim Paris, Ariel Rodriguez and Sheldon Chan for the excellent
`jimpic' hardware and software. As I had mentioned earlier
in this article, I find it to be the easiest way to get
started with PIC programming under Linux. Thanks to
Isaac Barona Martinez for uspp, which
simplifies serial programming a lot.
Conclusion
There are two excellent documents that describe serial
programming under Linux. One is the Serial
Programming HOWTO. The other is Serial
Programming guide for POSIX operating Systems. The
Microchip home page
contains lots of application notes, reading which might give
you ideas for your next Linux hardware hack - just don't
forget to share the fun with LG readers! I can be
contacted via my home page at pramode.net.
I am an instructor working for IC Software in Kerala, India. I would have loved
becoming an organic chemist, but I do the second best thing possible, which is
play with Linux and teach programming!
Object Caml is an ML type of language. For the non-gurus: it's a functional
language that can also be programmed in a non-functional and object-oriented way.
This language is really easy to learn. It's powerful and keeps impressing
me with its speed. Programs written in this language are almost always stable
by default. No segmentation faults, only occasional unending loops for the
programmers that still hang on to program their own loops. It is really not
needed to write most loops, since the libraries contain standard functions that
are good enough in 99% of the cases. So try to use those functions: It really
pays off in terms of stability of your programs, and, unless you have intimate
knowledge of the inner works of this language, they tend to be better
optimised.
The language can be obtained from the website caml.inria.fr. Here, they provide RPMs for the RedHat 7.2/8.0/9 and Mandrake 8.0 distributions. Also MS Windows binaries are available, but not all Unix library functions will work there, for some mysterious reason. The source tarball does compile flawlessly for me. It just has a somewhat unusual makefile layout:
# ./configure
# make world; make opt; make install
The normal libraries include many usable data-structures like balanced trees, hash tables, and streams.
Their version of header files (.mli files) contain all the basic documentation you need, and those are directly converted into HTML and published on the Web in their OCaml manual. This manual is not very usable to study this language, so I'll try to explain here some of the basic language constructions. This is just to give you an impression of the power of this language.
Modules & Functions
Now some real life examples. I wrote a program to help administrating a computer. It is a subset of a normal file finder, but is a command line tool and very fast. It helps locating large, not-recently-used files to be deleted from the system. It crawls through the directory tree and show the contents in different layouts.
Every module in OCaml has its own namespace. Specific definitions can be found by adding the module name, with the first character an upper-case character. You can also change the namespace of the current program to include a total module. Normally, only the standard module 'pervasives.mli' is included in the default namespace.
The example program 'show.ml' starts with:
This includes my own set of 'basics' functions and 2 standard libraries: 'Unix' and 'Unix.LargeFile'. A module normally consists of 2 files. The first file for exporting definitions 'module.mli' (like the C .h file), and the second one for actual code (the 'module.ml' file). The program uses the function 'string_sub' that provides a foolproof version of the 'String.sub' standard function (from the string.mli module).
The basics.mli file contains the lines:
val string_sub: string -> int -> int -> string
(** Get the sub string from a [string] from position [from] with [length].
This is the same function as String.sub, but it will never raise an exception.
And a negative [from] value is counted from the right side of the string. *)
This gives the definition of this function and the description. There is an automatic documentation generator (ocamldoc) that reads .mli files and writes .html files as basic interface documentation. Normal comments start with (* but the documentation generator only writes comments that start with (** to the .html files. This document contains links to the documentation of the used modules.
This documentation is really helpful to start programming ocaml. The .mli files are all included in the distribution, but the complete manual and a book can be downloaded from the Web site caml.inria.fr
The function is followed by its type. It wants 3 parameters and provides a string. Normally we need to write 'Basics.string_sub' to use this function. But after the 'open Basics' instruction just 'string_sub' is enough.
Basic operations and function calls
Now, back to the main program again. The first function is 'gettype'. It will try to return the type of a file. The file type is defined as the part of the filename following the last '.'. When there is no dot, the type is unknown and returned empty.
let gettype file =
try
let pos = String.rindex file '.' in
String.sub file (pos+1) (String.length file-pos-1)
with Not_found -> ""
;;
This function only uses standard functions. First, it catches the Not_found
exception in the 'try' 'with Not_found -> ""' code. All other exceptions will
be passed to the caller to be handled, and can possibly stop the main program.
The local variable pos get is filled with the result of the function rindex.
This function is also the reason to catch the exception; otherwise, the main
program might stop on the first found file with no '.' in it. Local variables
can be declared everywhere inside ocaml with 'let <variable> = <value> in
<code>'. After the completion of the given code, the variable is out of
scope and will be forgotten. The data will be passed to the garbage collector
to be removed from memory.
Function calls do normally use brackets. The function call to 'String.sub' gets 3 parameters the string 'file' the integer '(pos+1)' and the integer '(String.length file-pos-1)'.
The last parameter calls the function 'String.length' with a single parameter 'file'. So, the functions are eager for their parameters; brackets are needed only when the parameters are filled with calculations.
Also '(+)' and '(-)' are functions of the pervasives module. It is very easy to define your own operators; just add brackets around their definition, and they are ready.
If then else
The next routine 'filesize' in the example code is far longer, but largely introduces sub-functions and 'if <bool-expr> then <expr> else <expr>' statements.
This function creates a string from an int64 number for human readable file and directory sizes. The types of parameters are normally not given; they are determined by ocaml through their usage. When something is not clear, the compiler or interpreter will complain about it before executing the code.
let filesize s =
let tostr f =
if f>9.9 then
string_of_int (int_of_float (f +. 0.5))
else
let res = string_of_float (floor (f *. 10.0 +. 0.5) /. 10.0) in
if String.length res=2 then
res ^ "0"
else
res
in
let bytes = Int64.to_float s in
if bytes > 512.0 then
let kb = bytes /. 1024.0 in
if kb > 512.0 then
let mb = kb /. 1024.0 in
if mb > 512.0 then
let gb = mb /. 1024.0 in
tostr gb ^ " Gb"
else
tostr mb ^ " Mb"
else
tostr kb ^ " kb"
else
Int64.to_string s
;;
The ocaml standard library has a set of conversion functions. These functions normally follow the form of 'int_of_float' and 'string_of_float'. Specific types like 'Int64' use shorthand notations like 'Int64.to_float'. String concatenations are done with the operation '(^)'. Normally, functions are defined for only one specific type, so there are new sets of arithmetic functions for floats like '(+.)', '(*.)' and '(/.)'. The 'tostr' sub-function has some extra calculation to change something like '5. Gb' into the nicer form of '5.0 Gb'.
List notation and type conversion
The next function, 'converttime', converts a string into a float. OCaml uses floats for date for 2 reasons. The first is to prevent possible Year 2k problems, and can also be used for less than one-second time measurements. The function accepts English acronyms for month names. So let's introduce the list and the pair to create a translation of acronyms into numbers.
This list is totally static, and can be used easily by the standard function List.assoc to convert a string into the corresponding number.
let converttime str =
try
begin match
if str>"a" && str<"z" then
( int_of_string (string_sub str (String.rindex str ' '+1) 99),
List.assoc (string_sub str 0 3) month,
1
)
else
( int_of_string (string_sub str 0 (
try String.index str '-' with Not_found -> 99
)),
( try let pos=String.index str '-'+1 in
int_of_string (string_sub str pos (
try String.index_from str pos '-'-pos with err -> 99
))-1
with err -> 0
),
( try let pos=String.index str '-'+1 in
int_of_string (string_sub str (String.index_from str pos '-'+1) 99)
with err -> 1
)
)
with (yr,mn,md) ->
(* print_string ("Last access before: "^
string_of_int (if yr<50 then yr+2000 else if yr<100 then yr+1900 else yr)^"-"^
string_of_int (mn+1)^"-"^
string_of_int md^"\n");
*)
fst (mktime
{ tm_sec = 0; tm_min = 0; tm_hour = 0;
tm_mday = md; tm_mon = mn;
tm_year = if yr<50 then yr+100 else if yr<100 then yr else yr-1900;
tm_wday = 0; tm_yday = 0; tm_isdst = false
})
end with err ->
print_string ("Cannot decipher this date string '" ^ str ^ "'\n"); max_float
;;
The new operation in this function is the 'match <expr> with <template> -> expr'. This is one of the most versatile instructions of ocaml. It can be used to examine the contents of variables and get the needed information out of it. This function creates the triplet (year, month, day-of-month) out of 2 different date notations.
To debug this function the 'print_string' instruction is included but commented out to prevent clutter in the output of the program. Normally there is some logging mechanism to make the extra messages optional for the user.
The 'print_string' shows the ISO notation of the given date; it creates a 4-digits year and gives a month number with January=1 instead of the internal Unix use of January=0.
This function also shows the use of 'try <expr> with err -> <expr>' that caches every possible exception and fills the variable 'err' with the details of the exception. This function can raise quite a lot of different exceptions, and frankly I am not very interested in the details. The routine just complains to the user about the given date string and gets over it. It returns the maximal possible float to include every filename.
The main standard function is the 'Unix.mktime' function. It wants to get a record filled with numbers about the current time. This function returns a pair with the needed float and a normalized record. With the pervasives function fst returns just the first parameter of the pair.
The ';' before the 'max_float' indicates that the expression results in a float, but the instructions before the ';' are calculated first. This is the first non-functional instruction inside the example code. OCaml is not strictly functional, but has the full power of other functional languages.
Dynamic data structure
Now is the time for a real data structure that is dynamically build and can be used in a lot of different ways.
type entrytype =
| Dir of entry list (* directory with a list of files *)
| File of string (* a file inside a directory *)
and
entry = {
mutable e_name: string; (* name of a file or directory *)
e_type: entrytype; (* what type is this together with type
related information *)
e_atime: float; (* last access time *)
e_size: int64; (* size of the file or size of all the matching
files in the directory *)
}
The 'and' statement is used to glue the two definitions together. They are created at the same time so that 'entrytype' can include 'entry' and vice-versa. 'entrytype' can consist one of 2 things: a directory with a list of entries or a file with its type. The directory entry has a mutable name. This is can be used later on to change a filename info the full path to that file.
As with ANSI C, the operators for Boolean algebra are '(&&)' and '(||)'.
Recursion
let rec dirwrite el depth sortfn =
List.iter (
fun e ->
match e.e_type with
| Dir lst ->
if e.e_size <> Int64.of_int 0 then begin
print_string ((String.make (depth*2) ' ') ^ "Directory " ^
e.e_name ^ " = (" ^ filesize e.e_size ^ ")\n");
dirwrite lst (depth+1) sortfn
end
| File string ->
print_string ((String.make (depth*2) ' ') ^ e.e_name ^
" (" ^ filesize e.e_size ^ ")\n")
) (List.sort sortfn el)
;;
Here is the recursive ('rec') function 'dirwrite' that traverses a given tree 'el' and writes the result to the standard output. The parameter 'depth' indicates the amount of spaces to write a tree like structure of filenames. The function sorts all the lists with the given function 'sortfn'.
The new language structure here is 'fun <parm-1> ... <parm-n> -> <expr>'. This construction creates a function without a name. The parameters of this function like construction can be used like a template to match pairs.
This function suppresses directories that are 0 bytes in size to reduce clutter.
Variables vs. definitions
(* List of global variables *)
let min_size = ref (Int64.of_int 0) and (* minimum size of a file in bytes *)
last_access = ref max_float and (* last access time in seconds since 1970 *)
has_type = ref "" and (* type of file to show or empty to
show all *)
name_match = ref "" and (* regular expression to match the filename
with; empty is show all *)
name_regexp = ref (Str.regexp "") and (* pre-calculated regular expression *)
no_symlinks = ref false (* don't follow symbolic links to
directories *)
;;
This is a list of variables that can be changed due to the 'ref <expr>' construction. Normally definitions are just a label to their contents. These definitions are pointers to the memory and can be read by '!<variable>' and written by '<variable> := <expr>'. The parameters given to the program can make changes to the way the files are read.
let rec dirread path =
let list = ref [] and
size = ref (Int64.of_int 0) in
try
let dh = opendir path in
while true do
let file = readdir dh in
if file<>".." && file<>"." && file<>"CVS" && String.sub file 0 1 <> "." then
let s=stat (path^"/"^file) in
if s.st_kind = S_DIR &&
(not !no_symlinks || (lstat (path^"/"^file)).st_kind <> S_LNK)
then
let dir = dirread (path^"/"^file) in
list :=
{ e_name = file;
e_type = Dir (fst dir);
e_atime = s.st_atime;
e_size = snd dir
} :: !list;
size := Int64.add !size (snd dir)
else if
(!has_type = "" || gettype file = !has_type) &&
s.st_size > !min_size &&
s.st_atime < !last_access &&
(!name_match = "" || Str.string_match !name_regexp file 0)
then begin
list :=
{ e_name = file;
e_type = File (gettype file);
e_atime = s.st_atime;
e_size = s.st_size;
} :: !list;
size := Int64.add !size s.st_size
end
done;
(!list, !size)
with
| End_of_file -> (!list, !size)
| Unix_error (EACCES, err, parm) -> (!list, !size)
;;
The following functions are introduced in the function 'dirread':
Unix.opendir to start reading a directory.
Unix.readdir to read a filename.
Unix.stat for a record (Unix.stats) of statistics on a file.
Unix.lstat for statistics on a link.
Int64.add to add two int64 type of variables
Str.regexp to create a new interpreted regular expression
Str.string_match to match a string against a regular expression
Pervasives.(::) to create a list with an extra element in front of the old one
Pervasives.true as a Boolean constant
Pervasives.snd to return the second part of a pair
exception Unix.Unix_error (EACCESS, err, parm) that is raised when an access denied is encountered.
There is also a new construction 'while <boolean-expr> do <code> done' it just does what it is supposed to do.
Small is beautiful
let rec flat el path =
List.fold_right (
fun e ls ->
match e.e_type with
| Dir lst -> flat lst (path ^ "/" ^ e.e_name) @ ls
| File string ->
e.e_name <- (path ^ "/" ^ e.e_name);
e :: ls
) el []
;;
This neat little routine 'flat' hits the tree 'el' flat on the ground. It takes every file from every branch and creates a single list of all the encountered files. This is done with one of the most versatile standard routines inside ocaml: the 'List.fold_right' routine. This routine introduces a state machine (scarab) that crawls over a list and operates on every encountered element. It produces a new structure (droppings) as a result -- in this case, a flattened list.
The construction '<record-field> <- <expr>' changes the contents of a mutable record field. Without mutable fields, you can mutate records only by creating a new one with lots of fields inherited from the old one. This is a shortcut for that.
let name_order a b =
compare a.e_name b.e_name
;;
let type_order a b =
let typea = match a.e_type with Dir ls -> "dir" | File tp -> tp and
typeb = match b.e_type with Dir ls -> "dir" | File tp -> tp in
if compare typea typeb = 0 then
compare a.e_name b.e_name
else compare typea typeb
;;
let atime_order a b =
compare a.e_atime b.e_atime
;;
A set of sorting functions to use inside 'dirwrite'. The function 'compare' results in the widely used values of -1 for lower than, 0 for equal and +1 for higher than.
Command line parameters
let dir = ref "." and
sort = ref name_order and
show = ref 0
in
Arg.parse [
("-t",Arg.Unit (fun () -> sort := type_order),
"Sort by type and filename");
("-l",Arg.Unit (fun () -> sort := atime_order),
"Sort by last access time");
("-n",Arg.Unit (fun () -> show := 1),
"List filenames");
("-b",Arg.Unit (fun () -> show := 2),
"List both filenames and sizes");
("-s",Arg.Unit (fun () -> no_symlinks := true),
"Don't follow symbolic links");
("--before",Arg.String (fun s -> last_access := converttime s),
"Last access older than give date (format 'yyyy-mm-dd' or 'mmm yyyy')");
("--size",Arg.Int (fun i ->
min_size := Int64.mul (Int64.of_int i) (Int64.of_int (1024*1024))
), "File size bigger than size in Mbytes");
("--type",Arg.String (fun s -> has_type := s),
"File is specific type");
("--name",Arg.String (fun s ->
name_match := s; name_regexp := Str.regexp (s ^ "$")
), "Filename matches regular expression")
] (fun d -> dir := d) "show [DIR]";
let res = dirread !dir in
if !show=0 then begin
dirwrite (fst res) 0 !sort;
print_string ("Total size " ^ filesize (snd res) ^ "\n")
end else
List.iter
(fun e ->
print_endline (e.e_name ^ if !show=2 then " ("^filesize e.e_size^")" else "")
) (List.sort !sort (flat (fst res) !dir))
;;
And here is the main routine. It calls the Arg.parse routine to parse the parameters given to the program. But this is too much un-GNU for me. I wrote my own version of it that follows the GNU coding standards a bit more than the default one (Gnuarg). The other version is a bit more complicated so I will include only the sources that use it.
Generating binaries
The code can be obtained from here. Just unpack it somewhere with 'tar -xzf show.tar.gz' and move into the source directory with 'cd show/src'.
There is also a Makefile that compiles to machine code and installs everything. But Makefiles are too rough for sour eyes to show in this article. The nitty-gritty details are there in the source. The general compile form is.
ocamlopt -o show unix.cmxa str.cmxa basics.cmx show.ml
The only non-standard libraries in use here are unix.cmxa and str.cmxa.
make
su
make install
exit
show --help
show -s ~ --size 3 --before "apr 2003"
That concludes this example program.
Language features
Garbage collector
Just forget variables that contain complete data structures. Once it gets out of scope, the total structure will be eliminated from memory in due time.
Flexible data-structures
Any 2 data structures can be combined without hassle. Just create an array of records that contain 2 fields with hash tables of strings. No problem there... everything in a single variable than can be passed to functions or can be used globally in the program.
No pointers needed
A variable can have any type and when a new variable is created
Flexible in language boundary checks
The language can check array and string boundaries automatically, or those checks can be turned off for an extra speed boost. Without it, the language can give a segmentation fault, but that is the programmer's choice.
High quality error handling
Totally integrated into the language and no notable performance hit.
Native code generator and byte code interpreter
All the tools are there -- interpreter (ocaml), byte code (ocamlc) and native code compiler (ocamlopt) -- every wish is granted. The package comes also with a documentation generator (ocamldoc) and a simple to use profiler (ocamlprof) that adds usage counts as comments to the original source code. The language is also compatible with the more sophisticated profilers around.
ANSI-C compatibility layer
It is possible to include ANSI C routines inside OCaml programs, and OCaml routines inside C programs. This has a very easy to use API. Slightly less easy is the creation of OCaml data structures inside C; for me, that was the source of many segfaults. So, my routines call exported OCaml routines to fill data structures and create only OCaml strings and numbers in C. That way I won't have the hassle to debug the C code... OCaml is much easier to debug for me.
Object orientation
Not my favourite programming paradigm, but it is possible to build object-oriented programs in this language. Those features are not part of this article. I can live without them.
An active mailing list
This list is at caml-list@inria.fr and is normally in English. Yes, this originally French project has taken the burden to translate almost everything they got. This is no easy feat for them, so be grateful.
Cons:
Duplicate efforts in libraries
There are separate libraries for different type of big arrays, big files, and extra long integers. This isn't a big problem, because you can always just start with the normal structures and drop in the special library when need arises. The naming of the different functions is very much standardized, so renaming of function calls isn't needed much. The extra long integers though are too much different from normal integers. That part of the standard functions really need some tuning.
Readability
You need to be familiar with the basis constructions of the language, to make any sense of the actual code. Some constructions can look really weird without intimate knowledge of the language. OCaml is not a very natural language and has a very powerful, short notation for things. But this not much worse than languages like ANSI C, Perl, or lisp.
Not known enough in the Linux world
This language has excellent interfaces to standard libraries and easy binding to ANSI C, but still isn't very known. I like to create some articles like this to change that a bit. This is a really great language to program in, and gives you real power without the pitfalls common in other languages. Programmers should give it a try and feel that power once.
Developer at a small technology firm in the Netherlands called V&S bv.
(www.v-s.nl)
We sell firewall, anti-virus and spam boxes based on the Linux OS.
Using more and more the OCaml language to write my applications.
Busy writing a lightweight http server with an internal scripting language
(camlserv.sourceforge.net,
source code here)
Interested in writing AI based computer games. Always trying writing
one, nothing ready yet.