piątek, 7 grudnia 2012

Garrett D'Amore: [developer] webrev: pfile postmortem support

UPDATE:
Just noticed in repository that this work has been integrated:

changeset:   13875:f128a109e6d2
user:        Garrett D'Amore
date:        Fri Nov 02 09:48:42 2012 -0700
summary:     3294 pfiles postmortem support

Sorry that it goes through my blog, but I can't work through listbox weird URLs.
This is bit dated, as the thread started back in October. There is an interesting thread going just after this mail with reviews, but if you want to read it, go to listbox archive and search for "webrev: pfile postmortem support".
Below a quote from Garrett's mail that explains everything cleanly:

From: Garrett D'Amore 
Subject: [developer] webrev: pfile postmortem support
To: "developer@lists.illumos.org" 

http://cr.illumos.org/~gdamore/pfiles/

This adds the ability to use pfiles to get file information on a core dump.=
  To support this, I've had to create a new elf note, NT_FDINFO, and a stru=
cture for it (prfdinfo_t), which contains a bunch of information about open=
 file descriptors.  I've also added support for dumping this note into elfd=
ump.

This is the result of work I began at the illumos hackathon earlier in the =
month, and I've finally had time to get it to the point where I think its r=
eady for people to try, and also review it.  I think its fairly fully baked=
.  It works with both core files generate via gcore, and kernel generated c=
ore files.  (The two core dump generators are totally different code, addin=
g substantially to the effort.  Sadly I've had to teach elfexec far more ab=
out the kernel's user and file structures to support this than I'd like =85=
 the lack of good interfaces to those structures is an area of weakness in =
our current kernel subsystems.)

One thing that this doesn't do is go into more detail about sockets, doors,=
 etc.  I figure that would be a future RFE to add yet additional note secti=
ons (or possibly extending the NT_FDINFO with more information).  I think e=
ven without that information, the current data (which contains details abou=
t path, mode bits, open flags, etc.) is still very helpful for folks doing =
post-mortem debugging.  In particular, this is kernel state that is typical=
ly not available to folks debugging with traditional tools like gdb, dbx, o=
r even mdb.  (A cool follow on project might be to give those debuggers the=
 ability to parse this new note as well.  I'll leave it for someone more fa=
miliar with those tools.)

I've tested this pretty well on x86 systems, but I've not really tested thi=
s on SPARC.  Full testing should involve applying the patch, rebooting, and=
 then generating and testing cores  as well as live procfs attach based tes=
ts for pfiles.  Also, the new elfdump utility should be tested on sparc as =
well.  I'd be happy to hear about any one who wants to take the patch file =
above, test it out on sparc, and let me know.

The review is kind of juicy -- about 900 lines or so.  I'd really like to g=
et this integrated before *too* long though, so I'd appreciate meaningful a=
nd timely review feedback.  If you are going to plan on reviewing it, pleas=
e let me know, and give me an estimate of how long you'd like to review it.=
  Thanks!

Also, any testing (esp. sparc, but not *just* sparc) would be gratefully re=
ceived.

 - Garrett



-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=3Dnow
RSS Feed: https://www.listbox.com/member/archive/rss/182179/22406080-a3f15b=
b4
Modify Your Subscription: https://www.listbox.com/member/?member_id=3D22406=
080&id_secret=3D22406080-849724b2
Powered by Listbox: http://www.listbox.com

czwartek, 22 marca 2012

joyent creates new filesystem for illumos

Yesterday estibi posted a link on irc:
https://github.com/joyent/illumos-joyent/blob/master/usr/src/uts/common/fs/hyprlofs/hyprlofs_vfsops.c#L25
It points to a new filesystem for illumos, a hybrid between lofs (for mounting iso images and other virtual media  filesystem that allows you to make a subtree of one filesystem appear in another -- e.g. in the case of readonly mounts of /usr and /lib into zones - thnx Joshua for correcting me) and tmpfs (in RAM filesystem). Source code explains:
The purpose is to create a high performance name space for files on which
 applications will compute.
 I'm waiting eagerly for joyent to further explain and post example use. You rock, guys!

wtorek, 28 lutego 2012

illumos NTFS-3g kernel module again

Today I saw an e-mail from Jean-Pierre André. He is very active around illumos NTFS-3g kernel module. Please use and test.
Link to Jean-Pierre André's e-mail.
Content pasted below for convenient read.
If you'd like OpenIndiana to support NTFS out of the box, please comment on https://www.illumos.org/issues/1721

Hi,

I am now releasing the fuse kernel module for OpenIndiana. Ntfs-3g
now fully passes the standard tests with this kernel module and without
need of the workarounds I had to insert earlier to cope with the bad
behavior of the fuse kernel module originated from OpenSolaris.

A few optimizations would still be useful. I might have a look at them
if there is enough demand.

Available on http://b.andre.pagesperso-orange.fr/openindiana-ntfs-3g.html
are three packages ready for use :

- a full ntfs-3g package in 32-bit mode with the fuse-lite library
  and ntfsprogs
- a full ntfs-3g package in 64-bit mode with the fuse-lite library
  and ntfsprogs
- a fuse kernel module package for both the 32-bit and 64-bit modes

The raw source files (not packaged according to OpenIndiana
standards) are also available there.

I am keeping these files available on line for some time. I also keep
the change sets available to whoever enters the source code of the
fuse kernel module into a public source code management
repository.

Enjoy,

Jean-Pierre

poniedziałek, 13 lutego 2012

illumian - debian-like distribution of illumos

Recently my attention got caught by illumian, a Debian-like distribution of illumos. Since I come from Debian world, having administered Debian based servers for over ten years now, this distribution that uses .deb package format caught my eye. While I slowly get used to IPS, I find it far less friendly than apt- and friends.
What is this illumian distribution? It's basically OpenIndiana with IPS exchanged in favor of .deb packages. ISO file that you can grab from there contains installation media for server solution, which means that if you want to get graphical environment working, you have to install X and GNOME packages yourself. In short it comes down to installing x11 and gnome packages and enabling gdm service via svcadm.
Have a nice testing. :)

wtorek, 7 lutego 2012

Experiences Starting an OpenSource Operating System. Garrett D'Amore speaks at SCALE10x about the genesis of illumos. Recorded by Deirdré Straughan on her Youtube channel.

środa, 2 listopada 2011

OpenIndiana and ntfs-3g

Update:
I have patch ready to change unhelpful on OpenIndiana, SmartOS and other illumos distributions mention of modprobe to suggest:
/usr/sbin/add_drv -m 'fuse 0666 root sys' fuse
Still have to find out, why the heck link was not created in the first place. :/


----------------------------------
I recently had to use ntfs formatted USB drive under OpenIndiana and run into a small problem.
I shamelessly used gparted to find which device maps to my usb disk. Then had following session with ntfs-3g binary:

ntfs-3g /dev/dsk/c2t0d0p1 /mnt/
The disk contains an unclean file system (0, 0).
The file system wasn't safely closed on Windows. Fixing.
fuse: device not found, try 'modprobe fuse' first

Both fuse and ntfs-3g are installed from repositories:
sfe                                   origin   online   http://pkg.openindiana.org/sfe/
sfe-encumbered                        origin   online   http://pkg.openindiana.org/sfe-encumbered/


After some digging around and asking on IRC I decided that either no one had this problem or no one bothered to post solution. modinfo showed me that fuse module is loaded, after all:

258 fffffffff7edd000   e680 290   1  fuse (fuse driver)
258 fffffffff7edd000   e680  28   1  fuse (filesystem for fuse)


However ntfs-3g itself didn't say anything useful. It also suggested to use modproble, which is just ridiculous on OpenIndiana. I decided to truss it.


open64("/dev/fuse", O_RDWR)       Err#2 ENOENT


One clue closed I downloaded spec file for fuse from http://pkgbuild.sourceforge.net/spec-files-extra/ and checked for %post step:

%post
( retval=0;
  /usr/sbin/add_drv -m 'fuse 0666 root sys' fuse || retval=1;
  [ "$retval" = 0 ] && ln -s /devices/pseudo/fuse@0:fuse /dev/fuse || retval=1;
  exit $retval
)


I did check if /devices/pseudo/fuse exists (it did) and then created /dev/fuse link to it. Then ntfs-3g worked like a charm.

Cheers!

ZFS is ten

ZFS turned ten years old. There is short but nice blog post about it. One thing worth noting is, how long it takes  a feature from initial implementation to first introduction to operating system. Four years before inclusion in OpenSolaris project and then in Solaris 10 in 2006 IIRC.

http://blog.delphix.com/matt/2011/11/01/zfs-10-year-anniversary/

wtorek, 13 września 2011

OpenIndiana 151a release

OpenIndiana project has just released stable 151a release.
Follow download link for the goodness. There are also instructions for upgrading from previous OpenIndiana releases and from OpenSolaris.

środa, 23 lutego 2011

illumos based OpenIndiana

Update 26.02.2011.
There is also a USB image. You can use it, quite conveniently, with dd tool on Linux or with OpenSolaris USB Live creator on Windows written some time ago by Hiroshi Chonan.
To write it to USB stick with dd, you have to prepend a special header, which you can download here. Just do:
wget http://dlc.openindiana.org/isos/2G.header
wget http://dlc.openindiana.org/isos/148/oi-dev-148a-x86-20110223-1.usb
cat 2G.header oi-dev-148a-x86-20110223-1.usb >> oi-dev-148a-x86-20110223-1-header-prepended.usb
dd if=./oi-dev-148a-x86-20110223-1-header-prepended.usb of=/path/to/your/usb/device (ie. /dev/sdb)


Of course, with OpenSolaris based distributions, you can use /usr/bin/usbcopy from pkg:/install/distribution-constructor

Cheers!
-------------------
And so it happened. OpenIndiana has now published illumos based authority and isos.
Authority: http://pkg.openindiana.org/dev-il/ - based on illumos changeset fe7962c08d1d; the latest changeset in illumos repository.
Isos: http://dlc.openindiana.org/isos/148/oi-dev-148a-x86-20110223-1.iso

wtorek, 18 stycznia 2011

Illumost testing build

UPDATE: Installation instructions corrected. Thanx aszeszo for that. :)
Last week I've built packages from illumos sources, changeset cac385f011a3. Project OpenIndiana was kind to set up a repository with packages that you can use to upgrade your OpenIndiana system:

http://pkg.openindiana.org/illumos-experimental/
General instructions follow:

pkg set-publisher --non-sticky openindiana.org
pkg set-publisher -P -O http://pkg.openindiana.org/illumos-experimental/ on-nightly
pkg uninstall entire
pkg image-update -v --be-name illumos-1
Enjoy.


środa, 20 października 2010

find and symlinks

A pearl of wisdom from today problems: find does not automatically descend into symlinks to directories. You have to ask it to, by using -follow. It turns out that trivia is not widely known at this site.
This has today surfaced with filesystem hierarchy resembling below:

drwxr-xr-x /usr/local/bin/Testing
lrwxrwxrwx /opt/Testing -> /usr/local/bin/Testing


It translates into: /usr/locla/bin/Testing is a directory and /opt/Testing is a symbolic link to this directory. Within this directory lies a file test.t.
A developer was bewildered why, when he cds into /opt/Testing, following find the file:

find . -name test.t


and following does not:

find /opt/ -name test.t


Reason is simple. find, by default, will not follow symlinks to directories. There is an option for that: -follow.

poniedziałek, 13 września 2010

Closed crypto removed from illumos - FLAG DAY

Hi

I paste e-mail from Garrett, which constitutes a flag day for everyone building illumos nightly. Please, read it carefully, as it explains itself nicely:

With my integration of "6 Need open kcfd", there is actually no more
kcfd or signed crypto tarballs. (The necessary code for thread
management is in the kernel kcf. You'll notice new kernel LWPs called
"kcfpoold" in ps -eLf output.) Also, all vestiges of FIPS-140 support
are removed.

This represents a flag day for everyone building illumos.

You must do a full nightly across thsi flag day -- incrementals *will*
break.

You must not have a crypto tarball. (One less thing to download from
Oracle.. yay!)

You *MUST* use the version of nightly in the latest source. It is a
syntax error to try to use the old nightly -- it won't work unless you
have the old crypto tarball, and if you do, the result will not be
functional!

To recap:

* remove crypto tarball if present.
* use nightly from source tree
* do a full clobber build.

Thank you.

- Garrett

How to build illumos guide has been updated to follow this change. Enjoy!

środa, 1 września 2010

Broken splashimage with illumos boot menu

Due to some misunderstanding, a solaris.xpm have been removed too early from illumos source tree. It results in problems while trying to boot your new illumos boot environment, if you have updated with onu.sh script.
If you happen to have this problem, what you need is modify your grub menu. Mine looks like this.

title os-devel_145
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/os-devel_145
splashimage /boot/solaris.xpm
foreground d25f00
background 115d93
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=graphics
module$ /platform/i86pc/$ISADIR/boot_archive

What you need to do, is edit it, so it looks like below:

title os-devel_145
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/os-devel_145
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

On somewhat related note, booting to single user mode is done with adding -s at the end of the kernel$ line:

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS -s

The problem with missing splashimage is known and a bug has been filed.

niedziela, 29 sierpnia 2010

Illumos building instructions

As far as we know, Oracle has stopped developing OpenSolaris in the open. For those of us, who would rather have Opensolaris become a real openly developed operating system, ptoject illumos came to life. Thanks to Garrett d'Amore et consortes. On the illumos wiki page there is a nice instruction on compiling your own post build 134 illumos source.
Already much work has been put in the source tree to liberate it from dependency on /extra repository and new putbacks arrive all the time.
If you'd like to check it for yourself, here is the link to How to build illumos guide.

wtorek, 15 czerwca 2010

Zones delegated administration

To counter my previous entry, this is something that touches Zones also, but is pretty nice feature:
http://static.opensolaris.org/on/flagdays/pages/20100607142839.html - delegated zones administration.
Basically a user or role in global zone can be designated in a zone configuration as a admin resource, thus giving the user/role additional abilities to control zones.
A good example taken from the link above:

           example# zonecfg -z myzone
zonecfg:myzone> add admin
zonecfg:myzone:admin> set user=zadmin
zonecfg:myzone:admin> set auths=login,manage,clonefrom
zonecfg:myzone:admin> end
zonecfg:myzone> commit


No, user zadmin can use profiled shell to administer zones.

Linux Brandz removed from ON

ONNV build 143 is a important for every one of us that uses lx Branded Zones - the Linux ones. There have been a stall in development since some time and forums are full of people looking for ways to install anything newer than CentOS images provided by the project. But since the build 143 Linux Brandz will be removed. If you you is too, like me, you have to plan for some migration.
And so Linux Brandz go away: http://static.opensolaris.org/on/flagdays/pages/20100612010403.html

poniedziałek, 7 czerwca 2010

SFE bootstrap script

Thanks to Tommw, who maintains sfe bootstrap script, you can set up a complete SFE compilation ready environment in few simple moves.
The script can be downloaded from here: http://tinyurl.com/bootstrap-sfe-latest-os20nn. Just use wget:

  1. wget -N "http://tinyurl.com/bootstrap-sfe-latest-os20nn"

  2. chmod a+rx bootstrap-sfe-latest-os20nn

  3. ./bootstrap-sfe-latest-os20nn username


Username is a login of a user that you'll use for compilation. The script does all the magic, just answer two or three simple questions - user's password.
Before you begin, you may want to:

  • Create a user that will be responsible for compilation. User should have a "Software Installation" profile assigned. I use my main user, who has both Software Installation and Primary Administrator profiles.

  • pfexec pkg install wget

Remember to send Thanks e-mail to address printed after the bootstrap script finishes.

środa, 28 kwietnia 2010

PSARC 2009/377; 2009/378; 2010/072

PSARC 2009/377
  • In-kernel pfexec implementation.

PSARC 2009/378
  • Basic File Privileges

PSARC 2010/072

  • RBAC update: user attrs from profiles,

  • 4912090 pfzsh(1) should exist,

  • 4912093 pfbash(1) should exist,

  • 4912096 pftcsh(1) should exist,

  • 6440298 Expand the basic privilege set in order to restrict file access,
  • 6859862 Move pfexec into the kernel, [that one is interesting - trochej]

  • 6919171 cred_t sidesteps kmem_debug; we need to be able to detect bad hold/

piątek, 14 sierpnia 2009

pfexec is a kind of sudo, right?

(Open)Solaris can cause some emotional turmoil in new users coming from Linux background. Its ifconfig works bit differently, load average is calculated differently, and so on, et caetera...
One of things that leave people rather puzzled is authorisation mechanism based on system roles: Role Based Access Control (RBAC). As always, you're encouraged to read fantastic docyumentation: System Administration Guide: Security Services, chapterł Roles, Rights Profiles, and Privileges. However, to make a very short introduction, RBAC defines roles and profiles that have assigned privileges (authorisations) regarding use of commands. Something like sudo, at the first sight. Those roles and profiles can be assigned to concrete users. Something like sudo. The first difference is simply cosmetic. Instead of invoking 'sudo' in front of privileged command, one invokes 'pfexec'. The second difference is strictly work based. RBAC (pfexec) comes with multitude of profiles and roles ready to be used. For last few years I used OpenSolaris I prepared new profile just once.
Third difference lays in profiled shells, ie. pfsh. If a user is assigned a profile and given a profiled shell, they will be able to invoke only commands which are assigned to said profile, but without using the pfexec keyword.
Fourth difference are roles. Roles are basically user accounts that cannot be logged into directly. One has to use su command to log into the roles. Also, user can only logged into the role which has been assigned to them. This is the default use of root account in OpenSolaris distribution.
Fifth difference is big. RBAC reaches deeply in structure of (Open)Solaris, down to special kernel structures. This, along with rbac programming libraries, allows to implement very fine grained security into programs written. So, a fragment of code can check, if invoking user has, say, authorisation to write to log files, and then do some action.

Anatomy.
    RBAC is built of three logic elements:
  • authorisations - users rights to use privileged functions,

  • execution profiles - a way of combining authorisations and command with attributes, ie. UID of command,

  • roles - accounts explained before

Authorisations are defined in a system file /etc/security/auth_attr.
Users get profiles, roles and authorisations assigned in a file /etc/user_attr. Some fun fact. Some time ago blogs and howtos instructed that to assign a role, profile or authorisation to user, one had to edit said file. However, it is officially discouraged way. Better way is to use commands. Commands are guaranteed not to break the file. It is important to understand, that broken profile in /etc/user_attr may render user unable to user any privileged commands, login to root account and so on.
Authorisation definitions are written in a file /etc/security/prof_attr.
And the most important, I think - definitions of privileged operations assigned to profiles are written in a file /etc/security/exec_attr.
I won't indulge in details, as those are well explained in documentation (man man, people!). I'll only mention, that commands assigned to security profile are listed in /etc/security/exec_attr. A bit mysterious is file /etc/security/auth_attr. When I studied RBAC I was puzzled. All other files are intertwined together and it is rather easy to trace how they work together, but I couldn't figure out this one, as I couldn't find any sign of listed there attributes any other place. It turns out, that this is the mechanism for implementing rbac in applications. A code can do a check against any authorisation listed in the file and if a user is assigned one, application may take certain actions.

Commands
Managing RBAC is nased on few commands.

  • roles - prints list of roles assigned to a user,

  • profiles - prints a list of users profiles. profiles -l prints authorisations and privileged commands of users every profile,

  • auths - prints list of users authorisations,

  • roleadd - creates a role, syntax similar to useradd,

  • rolemod - modifies a role,

  • useradd: option -P accepts list of profiles as its argument, option -R accepts list of roles, option -A accepts list of athorisations,

  • usermod: option -P accepts list of profiles as its argument, option -R accepts list of roles, option -A accepts list of athorisations.


WARNING!
rolemod and usermod overwrite assigned privileges. If one wants to assign a profile Software Installation to user that already has Primary Administrator, usermod line looks like that:

pfexec usermod -P "Primary Administrator,Software Installation" username

More information in manpages and : System Administration Guide: Security Services, chapter Roles, Rights Profiles, and Privileges.

czwartek, 15 maja 2008

beadm

In my last post I wrote a short howto to handmade zfs rpool cloning. It takes only a few steps, but forgetting one of them may provide wrong results. But OpenSolaris has one nice tool to do it all for you:

beadm create opensolaris-2

Simply do it, and your system will be cloned, your grub will have a new entry and the new filesystem's /etc/vfstab will have it's '/' entry corrected.
Now, why not do a clone every few days, just in case you ever remove your '/etc'?