Ticket #1971 (new bug)

Opened 4 years ago

Last modified 3 years ago

Oplib paths are truncated in PBC files

Reported by: rgrjr Owned by: dukeleto
Priority: normal Milestone:
Component: tools Version: trunk
Severity: medium Keywords: ops2c
Cc: Language:
Patch status: Platform:

Description

The problem seems to be that IMCC uses the correct path to load the oplib object file when compiling, but drops all directory information when producing the packfile, so that loading the PBC file fails to find the library.

1. Unpack the attached test case tarball, and edit the PARROT-HOME macro in the makefile to point to a local Parrot working copy.

2. Run "make" to build and run the test. Here's what the output looks like on my machine (GNU/Linux, openSUSE 11.3):

rogers@rgr> make /usr/src/parrot/ops2c --dynamic dynops/kea_cl.ops --quiet cc -I/usr/src/parrot/include -I/usr/src/parrot/include/pmc \

-o dynops/kea_cl_ops.o -c dynops/kea_cl_ops.c

ld -o dynops/kea_cl_ops.so dynops/kea_cl_ops.o -shared /usr/src/parrot/parrot -o test.pbc test.pir /usr/src/parrot/parrot -o aux.pbc aux.pir /usr/src/parrot/parrot test.pbc Could not load oplib `kea_cl_ops' current instr.: 'main' pc 0 (test.pir:4) make: *** [test] Error 1 rogers@rgr>

3. Apply parrot-lib-debugging.patch to the Parrot working copy, do "make parrot", and rerun the test:

rogers@rgr> make clean rm -f *.pbc *.dump *.dis dynops/kea_cl_ops.* rogers@rgr> make /usr/src/parrot/ops2c --dynamic dynops/kea_cl.ops --quiet Got path '/usr/src/parrot/runtime/parrot/dynext/os.so' for os cc -I/usr/src/parrot/include -I/usr/src/parrot/include/pmc \

-o dynops/kea_cl_ops.o -c dynops/kea_cl_ops.c

ld -o dynops/kea_cl_ops.so dynops/kea_cl_ops.o -shared /usr/src/parrot/parrot -o test.pbc test.pir Got path 'dynops/kea_cl_ops.so' for dynops/kea_cl_ops /usr/src/parrot/parrot -o aux.pbc aux.pir Got path 'dynops/kea_cl_ops.so' for dynops/kea_cl_ops /usr/src/parrot/parrot test.pbc Got path for kea_cl_ops Could not load oplib `kea_cl_ops' current instr.: 'main' pc 0 (test.pir:4) make: *** [test] Error 1 rogers@rgr>

This shows that IMCC is looking for "dynops/kea_cl_ops" (as you would expect), but the PBC file is looking for "kea_cl_ops" at load time and failing. Indeed, if you do "strings aux.pbc", you will find the latter but not the former. (As an aside, pbc_dump and pbc_disassemble do not work on aux.pbc. Unless you copy dynops/kea_cl_ops.so to runtime/parrot/dynext/ in the Parrot working copy, where it will be found.)

4. Change the aux.pir source to use the absolute pathname. This does not help:

rogers@rgr> make /usr/src/parrot/parrot -o aux.pbc aux.pir Got path '/home/rogers/projects/kea2/bug/dynops/kea_cl_ops.so' for /home/rogers/projects/kea2/bug/dynops/kea_cl_ops /usr/src/parrot/parrot test.pbc Got path for kea_cl_ops Could not load oplib `kea_cl_ops' current instr.: 'main' pc 0 (test.pir:4) make: *** [test] Error 1 rogers@rgr>

As before, just "kea_cl_ops" is stored in the PBC file.

FWIW, Parrot has never had this problem with dynamic PMC classes; the exact pathname used in the .loadlib directive makes it into the PBC file.

Attachments

bug-in-oplib-loading-110115.tgz Download (1.4 KB) - added by rgrjr 4 years ago.
Tarball to reproduce oplib loading problem

Change History

Changed 4 years ago by rgrjr

Tarball to reproduce oplib loading problem

follow-up: ↓ 2   Changed 4 years ago by plobsing

Setting PARROT-HOME to point to a parrot installation which used a --prefix fails. Please pull appropriate directory information out of parrot_config for your build purposes.

in reply to: ↑ 1 ; follow-up: ↓ 3   Changed 4 years ago by rgrjr

Replying to plobsing:

Setting PARROT-HOME to point to a parrot installation which used a --prefix fails. Please pull appropriate directory information out of parrot_config for your build purposes.

I don't get it. I'm not trying to load a Parrot oplib, but one of my own. How does using parrot_config address this problem?

in reply to: ↑ 2   Changed 4 years ago by plobsing

Replying to rgrjr:

Replying to plobsing:

Setting PARROT-HOME to point to a parrot installation which used a --prefix fails. Please pull appropriate directory information out of parrot_config for your build purposes.

I don't get it. I'm not trying to load a Parrot oplib, but one of my own. How does using parrot_config address this problem?

Your code does not compile for any value of PARROT-HOME on my machine because the directory layout is different than that assumed by the makefile. These differences are abstracted-away by parrot_config.

  Changed 4 years ago by plobsing

  • owner set to dukeleto
  • keywords ops2c added
  • component changed from none to tools

With some slight (platform-specific) manglage of the makefile I was able to build the library. There are two problems manifesting here.

First, parrot does not look for libraries in $CWD, so the oplib will not be found until the dynext path is properly ammended. This can be accomplished by using the -X option on the command line. Another alternative would be installing the oplib, which will also resolve this issue.

# example of working dynops
> parrot -X ./dynops test.pbc
0.6
It runs.

Second, the path to the oplib appears to not be preserved, only the filename. This is actually an ops2c bug. The name preserved in the bytecode is the name by which the library identifies itself. If the oplib is not going to exist at the root of one of the dynext directories, the oplib should have this information in its metadata (which it currently does not).

# example that should work
> parrot -X . test.pbc
Could not load oplib `kea_cl_ops'
current instr.: 'main' pc 0 (test.pir:4)

follow-up: ↓ 6   Changed 4 years ago by rgrjr

The statement "parrot does not look for libraries in $CWD" is not true for PMC libraries; I've been doing the equivalent of

	loadlib $P1, "dynclasses/character"

since mid-2005. I've just now started trying to use an oplib, so that is why I am surprised that they work differently. It seems to me that one behavior or the other is a bug.

in reply to: ↑ 5 ; follow-up: ↓ 7   Changed 4 years ago by plobsing

Replying to rgrjr:

The statement "parrot does not look for libraries in $CWD" is not true for PMC libraries; I've been doing the equivalent of {{{ loadlib $P1, "dynclasses/character" }}} since mid-2005. I've just now started trying to use an oplib, so that is why I am surprised that they work differently. It seems to me that one behavior or the other is a bug.

I was unable to build kea-cl (extreme makefile manglage required, doesn't build with sbcl as packaged in arch linux extra repo), as checked out of  https://rgrjr.com/svn/kea-cl/trunk/ (btw, if this is the new home for kea-cl, please update the Languages page on the wiki), so I cannot know for certain exactly what you are doing. However, let me assure you that the line of code you have provided most likely does not work as intended under Parrot HEAD.

If using a .loadlib directive, it should give an error give an error like

error:imcc:loadlib directive could not find library `foo_group'
	in file 'asdf.pir' line 1

If using the loadlib op, it should give Undef as a result.

in reply to: ↑ 6 ; follow-up: ↓ 8   Changed 4 years ago by rgrjr

Replying to plobsing:

I was unable to build kea-cl (extreme makefile manglage required, doesn't build with sbcl as packaged in arch linux extra repo) ...

I'm sorry you went to all this trouble; Kea-CL hasn't built with SBCL for at least 4 or 5 years now. And since I'm the only one who ever builds it, and only build it on one system, I haven't needed much of a configurator.

as checked out of  https://rgrjr.com/svn/kea-cl/trunk/ (btw, if this is the new home for kea-cl, please update the Languages page on the wiki) ...

I will, if I can get it to work again. Kea-CL has been dormant for over two years now, largely because I can't keep up.

so I cannot know for certain exactly what you are doing. However, let me assure you that the line of code you have provided most likely does not work as intended under Parrot HEAD.

Consider the following test script:

    .HLL "lisp"
    .loadlib "dynclasses/rational"

    .sub main :main
	    ## initial sanity checking.
	    $P0 = get_class "Rational"
	    unless null $P0 goto load2
	    print "Bug:  Couldn't find 'Rational' class.\n"
	    die 5, 1
    load2:
	    say "Works."
    .end

Given this script in a suitable place, here's all you need to prove that the .loadlib line does work as I intended:

    rogers@rgr> mkdir test
    rogers@rgr> cd test
    rogers@rgr> mkdir dynclasses
    rogers@rgr> cp /usr/src/parrot/runtime/parrot/dynext/rational.so dynclasses/
    rogers@rgr> /usr/src/parrot/parrot test-load-dynpmc.pir
    Works.
    rogers@rgr> (cd ..; /usr/src/parrot/parrot test/test-load-dynpmc.pir)
    error:imcc:loadlib directive could not find library `dynclasses/rational'
	    in file 'test/test-load-dynpmc.pir' line 1
    rogers@rgr> 

Note that I copy rational.so from the Parrot working copy into a "dynclasses" directory in order to have something to load with a relative pathname. The "cd .." line proves that it is using the relative pathname to load it, and not just finding the original rational.so.

Again, I'm sorry you went to all that bother.

in reply to: ↑ 7 ; follow-up: ↓ 9   Changed 4 years ago by plobsing

Replying to rgrjr:

Replying to plobsing:

I was unable to build kea-cl (extreme makefile manglage required, doesn't build with sbcl as packaged in arch linux extra repo) ...

I'm sorry you went to all this trouble; Kea-CL hasn't built with SBCL for at least 4 or 5 years now. And since I'm the only one who ever builds it, and only build it on one system, I haven't needed much of a configurator.

No worries. I'd build with CMUCL if it were packaged for my platform (unfortunately not). I have SBCL and CLISP, how much work would I have to do to get either of those to work? I can also help with a configure system if you are interested.

as checked out of  https://rgrjr.com/svn/kea-cl/trunk/ (btw, if this is the new home for kea-cl, please update the Languages page on the wiki) ...

I will, if I can get it to work again. Kea-CL has been dormant for over two years now, largely because I can't keep up.

I'll try to lend a hand to get you up to speed. Hopefully that helps a bit.

so I cannot know for certain exactly what you are doing. However, let me assure you that the line of code you have provided most likely does not work as intended under Parrot HEAD.

Consider the following test script: {{{ .HLL "lisp" .loadlib "dynclasses/rational" .sub main :main ## initial sanity checking. $P0 = get_class "Rational" unless null $P0 goto load2 print "Bug: Couldn't find 'Rational' class.\n" die 5, 1 load2: say "Works." .end }}} Given this script in a suitable place, here's all you need to prove that the .loadlib line does work as I intended: {{{ rogers@rgr> mkdir test rogers@rgr> cd test rogers@rgr> mkdir dynclasses rogers@rgr> cp /usr/src/parrot/runtime/parrot/dynext/rational.so dynclasses/ rogers@rgr> /usr/src/parrot/parrot test-load-dynpmc.pir Works. rogers@rgr> (cd ..; /usr/src/parrot/parrot test/test-load-dynpmc.pir) error:imcc:loadlib directive could not find library `dynclasses/rational' in file 'test/test-load-dynpmc.pir' line 1 rogers@rgr> }}} Note that I copy rational.so from the Parrot working copy into a "dynclasses" directory in order to have something to load with a relative pathname. The "cd .." line proves that it is using the relative pathname to load it, and not just finding the original rational.so.

I stand corrected. Parrot does search in $CWD (indirectly, by just giving up and trying the path unmodified). However, due to the previously mentioned bug in ops2c, there is no way to have a dynoplib under a relative path (they must be in the root of one of the dynext search paths). Also, through a quirk of dlopen, paths that don't contain a slash get interpreted specially, so dumping the dynoplib directly in $CWD will not work either.

My recommendation, short term would be to either add to the dynext search path (for example, by using the -X option), or to use one of the existing, working search paths (eg: $CWD/dynext).

in reply to: ↑ 8   Changed 4 years ago by rgrjr

Replying to plobsing:

Replying to rgrjr: No worries. I'd build with CMUCL if it were packaged for my platform (unfortunately not). I have SBCL and CLISP, how much work would I have to do to get either of those to work?

I'm not sure; it's probably not trivial, but may not be all that hard. I gave up on SBCL compatibility when I realized that using certain CMUCL internals would speed development of the cross-compiler, and didn't bother to look for the SBCL equivalents. In order to run the compiler natively, I had to get rid of those nonportable hacks, but I'm pretty sure that others have crept in during the meantime.

I can also help with a configure system if you are interested.

I will, if I can get it to work again. Kea-CL has been dormant for over two years now, largely because I can't keep up.

I'll try to lend a hand to get you up to speed. Hopefully that helps a bit.

Thank you for your kind offers, but I'm still trying to decide whether or not I want to put the effort into keeping Kea-CL going.

I stand corrected. Parrot does search in $CWD (indirectly, by just giving up and trying the path unmodified). However, due to the previously mentioned bug in ops2c, there is no way to have a dynoplib under a relative path (they must be in the root of one of the dynext search paths). Also, through a quirk of dlopen, paths that don't contain a slash get interpreted specially, so dumping the dynoplib directly in $CWD will not work either.

Ah, I wondered why that didn't work. Mystery solved.

My recommendation, short term would be to either add to the dynext search path (for example, by using the -X option), or to use one of the existing, working search paths (eg: $CWD/dynext).

I think I'll do the former, since that is more than adequate for running my test suite. Thanks again.

  Changed 3 years ago by jkeenan

rgrjr,

Do you have continuing concerns in this area? If not, can we close the ticket -- or perhaps open a new one focusing on what still needs to be addressed?

Thank you very much.

kid51

Note: See TracTickets for help on using tickets.