8000 Compatibility with Allegro v.11 · Issue #381 · cffi/cffi · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Compatibility with Allegro v.11 #381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
RealSharpness opened this issue Apr 6, 2024 · 20 comments
Open

Compatibility with Allegro v.11 #381

RealSharpness opened this issue Apr 6, 2024 · 20 comments

Comments

@RealSharpness
Copy link

I use Common Lisp library lispbuilder-sdl which relies on CFFI.

My lisp implementation Allegro CL has updated to version 11 and their internal symbol FF-FUNCALL has changed to FF_FUNCALL. (Not certain, but I think this is because it can now be used in the browser.)

For reference, here is the lispbuilder-sdl code which calls FF-FUNCALL.

cffi/src/cffi-allegro.lisp

Line 260 in 621cb9d

`(system::ff-funcall

I am a bit out of my depth in this topic, but would it be possible to update CFFI so the new symbol FF_FUNCALL is used?
@phoe
Copy link
Contributor
phoe commented Apr 7, 2024

Sure, we'll need someone who can write a fixed implementation conditionalized on ACL version, and then then run all the tests for both old and new ACL. (Won't be me, since I'm not an ACL user; does Franz have any sort of user forum where this issue might have already been raised or where we can ask for a list of FFI-related changes that would speed things up on CFFI side?)

@RealSharpness
Copy link
Author

Thank you,

I have not found a user forum and believe Franz handles all matters directly with individual customers.

https://franz.com/support/documentation/11.0/foreign-functions.html#30-the-foreign-function-interface-specification
This section of v.11 documentation describes ACL's FFI which is mostly the same on all versions "except as noted." The changed symbol which produced this error however is an undocumented internal symbol.

I can try requesting anything useful that isn't covered in the documentation, please let me know. I'll provide whatever support I can.

@phoe
Copy link
Contributor
phoe commented Apr 8, 2024

Sure, please try linking them to this issue.

@RealSharpness
Copy link
Author

I sent a follow-up e-mail to this issue that Franz helped me identify but haven't received a response yet.

I'm wondering if there's any ACL user here who can help me with this problem. I suppose there isn't any great demand at this time but I would offer what compensation I can.

@LEHunter
Copy link
LEHunter commented May 2, 2024

Yes, I am an ACL user who just ran into the same problem. I hadn't yet figured out the new symbol though, so thanks. I don't have an old version to run the tests on, but I can write a fix conditionalized on ACL version, and then then run all the tests for new ACL.

@LEHunter
Copy link
LEHunter commented May 2, 2024 •
edited
Loading

Well, I can help you, but not do the heavy lifting for the update. Change these two functions in src/cffi-allegro.lisp

  (declare (ignore convention library))
  (multiple-value-bind (types fargs rettype)
      (foreign-funcall-type-and-args args)
    `(#+(version>= 11 0) system::ff_funcall
      #-(version>= 11 0) system::ff-funcall  
      (load-time-value (excl::determine-foreign-address
                        '(,name :language :c)
                        #-(version>= 8 1) ff::ep-flag-never-release
                        #+(version>= 8 1) ff::ep-flag-always-release
                        nil ; method-index
                        ))
      ;; arg types {'(:c-type lisp-type) argN}*
      ,@(mapcan (lambda (type arg)
                  `(',(allegro-type-pair type) ,arg))
                types fargs)
      ;; return type '(:c-type lisp-type)
      ',(allegro-type-pair rettype))))

and

(defmacro %foreign-funcall-pointer (ptr args &key convention)
  (declare (ignore convention))
  (multiple-value-bind (types fargs rettype)
      (foreign-funcall-type-and-args args)
    (with-unique-names (entry-vec)
      `(let ((,entry-vec (excl::make-entry-vec-boa)))
         (setf (aref ,entry-vec 1) ,ptr) ; set jump address
         (#+(version>= 11 0) system::ff_funcall
          #-(version>= 11 0) system::ff-funcall 
          ,entry-vec
          ;; arg types {'(:c-type lisp-type) argN}*
          ,@(mapcan (lambda (type arg)
                      `(',(allegro-type-pair type) ,arg))
                    types fargs)
          ;; return type '(:c-type lisp-type)
          ',(allegro-type-pair rettype))))))

The only changes are the conditionalized change to system::ff-funcall.

This works for me, but the reason I'm not making a pull request is that it fails a bunch of tests that I don't have time to track down (see below). This seems to work for me.

54 out of 342 total tests failed: FUNCALL.CHAR, FUNCALL.INT.1, 
   FUNCALL.INT.2, FUNCALL.LONG, FUNCALL.LONG-LONG, 
   FUNCALL.UNSIGNED-LONG-LONG, FUNCALL.FLOAT, FUNCALL.DOUBLE, 
   FUNCALL.STRING.1, FUNCALL.STRING.2, FUNCALL.STRING.3, 
   FUNCALL.VARARGS.NOSTDLIB, FUNCALL.VARARGS.CHAR, FUNCALL.VARARGS.INT, 
   FUNCALL.VARARGS.LONG, FUNCALL.VARARGS.DOUBLE, FUNCALL.VARARGS.STRING, 
   FUNCALL.DOUBLE26, FUNCALL.FLOAT26, FUNCALL.F-S-P.1, 
   FUNCALL.POINTER-NOT-NIL, TRANSLATE-CAMELCASE-NAME.TO-SYMBOL, 
   TRANSLATE-CAMELCASE-NAME.TO-SYMBOL-SPECIAL, 
   TRANSLATE-CAMELCASE-NAME.TO-STRING-SPECIAL, DEFCFUN.NOOP, 
   DEFCFUN.VARARGS.NOSTDLIB, DEFCFUN.VARARGS.CHAR, 
   DEFCFUN.VARARGS.SHORT, DEFCFUN.VARARGS.INT, DEFCFUN.VARARGS.LONG, 
   DEFCFUN.VARARGS.FLOAT, DEFCFUN.VARARGS.DOUBLE, 
   DEFCFUN.VARARGS.STRING, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.STRING-NOT-DOCSTRING, CALLBACKS.NIL-FOR-NULL, 
   CALLBACKS.FUNCALL.1, CALLBACKS.FUNCALL.2, CALLBACKS.BFF.1, 
   CALLBACKS.BFF.2, CALLBACKS.DOUBLE26.FUNCALL, CALLBACKS.FLOAT26, 
   CALLBACKS.FLOAT26.FUNCALL, FOREIGN-ALLOC.10, POINTERP.4, POINTERP.5, 
   POINTER-EQ.NON-POINTERS.1, POINTER-EQ.NON-POINTERS.2, 
   NULL-POINTER-P.NON-POINTER.2, FSBV.WFO, FSBV.MAKEPAIR.1, 
   FSBV.MAKEPAIR.2, MISC-TYPES.EXPAND.2.
49 unexpected failures: FUNCALL.CHAR, FUNCALL.INT.1, FUNCALL.INT.2, 
   FUNCALL.LONG, FUNCALL.LONG-LONG, FUNCALL.UNSIGNED-LONG-LONG, 
   FUNCALL.FLOAT, FUNCALL.DOUBLE, FUNCALL.STRING.1, FUNCALL.STRING.2, 
   FUNCALL.STRING.3, FUNCALL.VARARGS.NOSTDLIB, FUNCALL.VARARGS.CHAR, 
   FUNCALL.VARARGS.INT, FUNCALL.VARARGS.LONG, FUNCALL.VARARGS.DOUBLE, 
   FUNCALL.VARARGS.STRING, FUNCALL.DOUBLE26, FUNCALL.FLOAT26, 
   FUNCALL.F-S-P.1, FUNCA
8000
LL.POINTER-NOT-NIL, 
   TRANSLATE-CAMELCASE-NAME.TO-SYMBOL, 
   TRANSLATE-CAMELCASE-NAME.TO-SYMBOL-SPECIAL, 
   TRANSLATE-CAMELCASE-NAME.TO-STRING-SPECIAL, DEFCFUN.VARARGS.NOSTDLIB, 
   DEFCFUN.VARARGS.CHAR, DEFCFUN.VARARGS.SHORT, DEFCFUN.VARARGS.INT, 
   DEFCFUN.VARARGS.LONG, DEFCFUN.VARARGS.FLOAT, DEFCFUN.VARARGS.DOUBLE, 
   DEFCFUN.VARARGS.STRING, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.STRING-NOT-DOCSTRING, CALLBACKS.NIL-FOR-NULL, 
   CALLBACKS.FUNCALL.1, CALLBACKS.FUNCALL.2, CALLBACKS.BFF.2, 
   CALLBACKS.DOUBLE26.FUNCALL, CALLBACKS.FLOAT26, 
   CALLBACKS.FLOAT26.FUNCALL, FOREIGN-ALLOC.10, POINTERP.4, POINTERP.5, 
   POINTER-EQ.NON-POINTERS.1, POINTER-EQ.NON-POINTERS.2, 
   NULL-POINTER-P.NON-POINTER.2, MISC-TYPES.EXPAND.2.

Overall unexpected failures: 
  misc-types.expand.2
  null-pointer-p.non-pointer.2
  pointer-eq.non-pointers.2
  pointer-eq.non-pointers.1
  pointerp.5
  pointerp.4
  foreign-alloc.10
  callbacks.float26.funcall
  callbacks.float26
  callbacks.double26.funcall
  callbacks.bff.2
  callbacks.funcall.2
  callbacks.funcall.1
  callbacks.nil-for-null
  callbacks.string-not-docstring
  defcfun.bff.2
  defcfun.bff.1
  defcfun.varargs.string
  defcfun.varargs.double
  defcfun.varargs.float
  defcfun.varargs.long
  defcfun.varargs.int
  defcfun.varargs.short
  defcfun.varargs.char
  defcfun.varargs.nostdlib
  translate-camelcase-name.to-string-special
  translate-camelcase-name.to-symbol-special
  translate-camelcase-name.to-symbol
  funcall.pointer-not-nil
  funcall.f-s-p.1
  funcall.float26
  funcall.double26
  funcall.varargs.string
  funcall.varargs.double
  funcall.varargs.long
  funcall.varargs.int
  funcall.varargs.char
  funcall.varargs.nostdlib
  funcall.string.3
  funcall.string.2
  funcall.string.1
  funcall.double
  funcall.float
  funcall.unsigned-long-long
  funcall.long-long
  funcall.long
  funcall.int.2
  funcall.int.1
  funcall.char
  misc-types.expand.2
  null-pointer-p.non-pointer.2
  pointer-eq.non-pointers.2
  pointer-eq.non-pointers.1
  pointerp.5
  pointerp.4
  foreign-alloc.10
  callbacks.float26.funcall
  callbacks.float26
  callbacks.double26.funcall
  callbacks.bff.2
  callbacks.funcall.2
  callbacks.funcall.1
  callbacks.nil-for-null
  callbacks.string-not-docstring
  defcfun.bff.2
  defcfun.bff.1
  defcfun.varargs.string
  defcfun.varargs.double
  defcfun.varargs.float
  defcfun.varargs.long
  defcfun.varargs.int
  defcfun.varargs.short
  defcfun.varargs.char
  defcfun.varargs.nostdlib
  translate-camelcase-name.to-string-special
  translate-camelcase-name.to-symbol-special
  translate-camelcase-name.to-symbol
  funcall.pointer-not-nil
  funcall.f-s-p.1
  funcall.float26
  funcall.double26
  funcall.varargs.string
  funcall.varargs.double
  funcall.varargs.long
  funcall.varargs.int
  funcall.varargs.char
  funcall.varargs.nostdlib
  funcall.string.3
  funcall.string.2
  funcall.string.1
  funcall.double
  funcall.float
  funcall.unsigned-long-long
  funcall.long-long
  funcall.long
  funcall.int.2
  funcall.int.1
  funcall.char
#<asdf/lisp-action:test-op >
#<asdf/plan:sequential-plan @ #x300fb39b2>
cl-user(4):

These look mostly like warnings, but I don't have time to track down 54 failed tests when it seems to work for me. Hope this helps!

@LEHunter LEHunter mentioned this issue May 2, 2024
Open
@RealSharpness
Copy link
Author

@LEHunter Bless you, this fix works on my end too.

@luismbo
Copy link
Member
luismbo commented May 6, 2024

@LEHunter can you evaluate (cffi:foreign-funcall "toupper" :char (char-code #\a) :char)? (that's the body of the first failing test FUNCALL.CHAR). My guess is that system::ff_funcall has extra return values.

@LEHunter
Copy link
LEHunter commented May 6, 2024 via email

@dieggsy
Copy link
dieggsy commented Jun 20, 2024

For what it's worth, the failing tests above are what I get in ACL 11 without the conditionalized sys::ff_funcall. With it, I see the following:

30 out of 342 total tests failed: FUNCALL.VARARGS.NOSTDLIB, 
   FUNCALL.VARARGS.CHAR, FUNCALL.VARARGS.INT, FUNCALL.VARARGS.LONG, 
   FUNCALL.VARARGS.DOUBLE, FUNCALL.VARARGS.STRING, DEFCFUN.NOOP, 
   DEFCFUN.VARARGS.NOSTDLIB, DEFCFUN.VARARGS.CHAR, 
   DEFCFUN.VARARGS.SHORT, DEFCFUN.VARARGS.INT, DEFCFUN.VARARGS.LONG, 
   DEFCFUN.VARARGS.FLOAT, DEFCFUN.VARARGS.DOUBLE, 
   DEFCFUN.VARARGS.STRING, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.BFF.1, CALLBACKS.BFF.2, CALLBACKS.FLOAT26, 
   CALLBACKS.FLOAT26.FUNCALL, FOREIGN-ALLOC.10, POINTERP.4, POINTERP.5, 
   POINTER-EQ.NON-POINTERS.1, POINTER-EQ.NON-POINTERS.2, 
   NULL-POINTER-P.NON-POINTER.2, FSBV.WFO, FSBV.MAKEPAIR.1, 
   FSBV.MAKEPAIR.2.
25 unexpected failures: FUNCALL.VARARGS.NOSTDLIB, FUNCALL.VARARGS.CHAR, 
   FUNCALL.VARARGS.INT, FUNCALL.VARARGS.LONG, FUNCALL.VARARGS.DOUBLE, 
   FUNCALL.VARARGS.STRING, DEFCFUN.VARARGS.NOSTDLIB, 
   DEFCFUN.VARARGS.CHAR, DEFCFUN.VARARGS.SHORT, DEFCFUN.VARARGS.INT, 
   DEFCFUN.VARARGS.LONG, DEFCFUN.VARARGS.FLOAT, DEFCFUN.VARARGS.DOUBLE, 
   DEFCFUN.VARARGS.STRING, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.BFF.2, CALLBACKS.FLOAT26, CALLBACKS.FLOAT26.FUNCALL, 
   FOREIGN-ALLOC.10, POINTERP.4, POINTERP.5, POINTER-EQ.NON-POINTERS.1, 
   POINTER-EQ.NON-POINTERS.2, NULL-POINTER-P.NON-POINTER.2.

Overall unexpected failures: 
  NULL-POINTER-P.NON-POINTER.2
  POINTER-EQ.NON-POINTERS.2
  POINTER-EQ.NON-POINTERS.1
  POINTERP.5
  POINTERP.4
  FOREIGN-ALLOC.10
  CALLBACKS.FLOAT26.FUNCALL
  CALLBACKS.FLOAT26
  CALLBACKS.BFF.2
  DEFCFUN.BFF.2
  DEFCFUN.BFF.1
  DEFCFUN.VARARGS.STRING
  DEFCFUN.VARARGS.DOUBLE
  DEFCFUN.VARARGS.FLOAT
  DEFCFUN.VARARGS.LONG
  DEFCFUN.VARARGS.INT
  DEFCFUN.VARARGS.SHORT
  DEFCFUN.VARARGS.CHAR
  DEFCFUN.VARARGS.NOSTDLIB
  FUNCALL.VARARGS.STRING
  FUNCALL.VARARGS.DOUBLE
  FUNCALL.VARARGS.LONG
  FUNCALL.VARARGS.INT
  FUNCALL.VARARGS.CHAR
  FUNCALL.VARARGS.NOSTDLIB
  NULL-POINTER-P.NON-POINTER.2
  POINTER-EQ.NON-POINTERS.2
  POINTER-EQ.NON-POINTERS.1
  POINTERP.5
  POINTERP.4
  FOREIGN-ALLOC.10
  CALLBACKS.FLOAT26.FUNCALL
  CALLBACKS.FLOAT26
  CALLBACKS.BFF.2
  DEFCFUN.BFF.2
  DEFCFUN.BFF.1
  DEFCFUN.VARARGS.STRING
  DEFCFUN.VARARGS.DOUBLE
  DEFCFUN.VARARGS.FLOAT
  DEFCFUN.VARARGS.LONG
  DEFCFUN.VARARGS.INT
  DEFCFUN.VARARGS.SHORT
  DEFCFUN.VARARGS.CHAR
  DEFCFUN.VARARGS.NOSTDLIB
  FUNCALL.VARARGS.STRING
  FUNCALL.VARARGS.DOUBLE
  FUNCALL.VARARGS.LONG
  FUNCALL.VARARGS.INT
  FUNCALL.VARARGS.CHAR
  FUNCALL.VARARGS.NOSTDLIB
T

@dieggsy
Copy link
dieggsy commented Jun 20, 2024 •
edited
Loading

After a little more testing, it seems some of these fail in ACL 10.1 too. Here are the tests that fail on ACL11 but not on ACL10.1:

FUNCALL.VARARGS.CHAR 
FUNCALL.VARARGS.INT 
FUNCALL.VARARGS.LONG 
FUNCALL.VARARGS.STRING 
DEFCFUN.VARARGS.CHAR 
DEFCFUN.VARARGS.SHORT 
DEFCFUN.VARARGS.INT 
DEFCFUN.VARARGS.LONG 
DEFCFUN.VARARGS.STRING 
DEFCFUN.BFF.1 
DEFCFUN.BFF.2 
CALLBACKS.FLOAT26 
CALLBACKS.FLOAT26.FUNCALL

@dieggsy dieggsy mentioned this issue Oct 28, 2024
Merged
@digikar99
Copy link

Is there a reason why cffi:foreign-type is defined as ff:foreign-address and not ff:foreign-pointer for Allegro CL? (Ref..)

Section 3.1.1 of the documentation makes a distinction between ff:foreign-pointer and ff:foreign-address:

A foreign-pointer is an instance of the class foreign-pointer (see also make-foreign-pointer) that has a special slot (with accessor foreign-pointer-address) intended to point to raw data.

A foreign adress is either a Lisp integer, a Lisp vector, or a foreign-pointer instance.

The same distinction is made by Allegroc CL 10.1.

@sionescu
Copy link
Member

@digikar99 I believe that ff:foreign-pointer didn't exist when CFFI's Allegro backend was written.

@digikar99
Copy link

Uhm, okay, I can trace the decision to use integers as foreign pointers on Allegro CL to the very start of Allegro CL support. I also see that @luismbo experimented with ff:foreign-pointer even back then but decided to go with integers. I'm curious at this decision!

Later on, cffi:foreign-pointer was added as a type alias and that perhaps used ff:foreign-address for backward compatibility.

@luismbo
Copy link
Member
luismbo commented Mar 22, 2025 •
edited
Loading

It was a long time ago, but ISTR that it was the natural decision given that the SBCL backend also used SAPs rather than the higher level alien pointers and ff:alocate-fobject returns integers.

The main goal was to avoid consing, I suppose.

I agree that a better typed representation would be better, in general.

@digikar99
Copy link

Understood! I 8000 n that case, I will issue a PR if I get down to rewriting this part using ff:foreign-pointer.

@luismbo
Copy link
Member
luismbo commented Mar 23, 2025

Understood! In that case, I will issue a PR if I get down to rewriting this part using ff:foreign-pointer.

I'm not convinced switching to a CLOS representation for pointers is a good idea.

Won't that introduce unnecessary consing and overhead?

@digikar99
Copy link

CLOS representation

Okay, just checked. Indeed, while SBCL has sb-sys:system-area-pointer as builtin and has put in a fair amount of optimizations towards them, Allegro CL pointers are still CLOS classes. So, even the simple act of incrementing a pointer would introduce a generic function dispatch instead of a single line of assembly :/

Won't that introduce unnecessary consing and overhead?

I agree. Alright, plan to rewrite using ff:foreign-pointer abandoned!

@digikar99
Copy link

Though, given this from the documentation,

A foreign adress is either a Lisp integer, a Lisp vector, or a foreign-pointer instance.

Do we want to make cffi:foreign-pointer as an alias to ff:foreign-address as this commit had done? Or would it be nicer to restrict cffI:foreign-pointer to be an alias to integer alone?

@luismbo
Copy link
Member
luismbo commented Mar 27, 2025

We could argue for supporting multiple pointer representations (maybe that could enable easier interoperability between CFFI and other code using ACL's FFI) but some bits of CFFI-SYS clearly only work with the integer representation so the type is not accurate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@luismbo @sionescu @LEHunter @dieggsy @phoe @digikar99 @RealSharpness
0