You can create custom BSDFs in Arnold using the C++ API in ai_shader_bsdf.h by filling in the callbacks for your specific BSDF. You can wrap it in a closure like so:
AtBSDF* bsdf = AiBSDF(sg, rgb_weight, my_bsdf_methods, my_bsdf_data_size); sg->out.CLOSURE() = AtClosure(my_color, bsdf);
This will work from within C++ shaders, but obviously not OSL shaders. Unfortunately, I don't think it's possible at this time to exposure arbitrary closures to OSL code directly. However, you should be able to create a C++ shader that outputs a closure that can be connected as an input closure color to an OSL shader (and vice-versa) in the shader network.
Is this something that is planned? Having the ability to write BxDFs and expose them as OSL closures would be great.