[OTDev] How to upload a dataset ...

Mon Sep 13 11:19:00 CEST 2010

Hi,

It would be helpful to have a working sample of a short dataset in
RDF. That way I can compare this to my dataset representation to check
why the upload task is failing on the ambit server.

Regards
Surajit

On 10 September 2010 00:24, surajit ray <mr.surajit.ray at gmail.com> wrote:
> Hi,
>
> Tried changing the order of the statements, putting this
>
> dataset.addProperty(OT.dataEntry, dataEntry);
>
> right after all the feature and feature value creation steps. I am
> still getting the error on upload. Could you suggest the reason - so I
> can troubleshoot it at our end
>
> Thanks
> Surajit
>
> On 6 September 2010 12:27, Nina Jeliazkova <jeliazkova.nina at gmail.com> wrote:
>> Hi,
>>
>> On Fri, Sep 3, 2010 at 8:58 PM, surajit ray <mr.surajit.ray at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Lemme take a look into that ....
>>>
>>> heres my code creating the dataset ... can you tell me whats wrong with it
>>> ?
>>
>>
>>>
>>> *****************************************
>>>             this.resultDataset = createModel();
>>>             Individual dataset =
>>>
>>> resultDataset.createIndividual(resultDataset.getOntClass(OT.OTClass.Dataset.getNS()));
>>>             OT.OTClass.Dataset.createOntClass(resultDataset);
>>
>> Have to check if it indeed makes different to Jena, but may be worth
>> changing the order of the two statements above (first creating the dataset
>> class and then using it)
>>
>>>
>>>             OT.OTClass.DataEntry.createOntClass(resultDataset);
>>>             OT.OTClass.Feature.createOntClass(resultDataset);
>>>             OT.OTClass.FeatureValue.createOntClass(resultDataset);
>>>             OT.OTClass.Compound.createOntClass(resultDataset);
>>>
>>>
>>>             for (int i=0 ; i<this.compoundURIs.length ; i++) {
>>>                 //create data entry
>>>                 Individual dataEntry =
>>>
>>> resultDataset.createIndividual(OT.OTClass.DataEntry.getOntClass(resultDataset));
>>>                 dataset.addProperty(OT.dataEntry, dataEntry);
>>>                 //add compound to data entry
>>>                 Individual newcompound =
>>> resultDataset.createIndividual(this.compoundURIs[i],
>>> OT.OTClass.Compound.getOntClass(resultDataset));
>>>                 dataEntry.addProperty(OT.compound, newcompound);
>>>                 //First value
>>>                 Individual feature1 =
>>>
>>> resultDataset.createIndividual(MaxtoxApplicationSettings.getServerRootPath()
>>> + "/feature/" + modelNumber + "_prediction",
>>> OT.OTClass.Feature.getOntClass(resultDataset));
>>>                 Individual featureValue1 =
>>>
>>> resultDataset.createIndividual(OT.OTClass.FeatureValue.getOntClass(resultDataset));
>>>                 featureValue1.addProperty(OT.feature, feature1);
>>>                 featureValue1.addLiteral(OT.value,
>>> resultDataset.createTypedLiteral(this.predictions[i],
>>> XSDDatatype.XSDstring));
>>>
>>>                 //Second value
>>>                 Individual feature2 =
>>>
>>> resultDataset.createIndividual(MaxtoxApplicationSettings.getServerRootPath()
>>> + "/feature/" + modelNumber + "_fingerprint",
>>> OT.OTClass.Feature.getOntClass(resultDataset));
>>>                 Individual featureValue2 =
>>>
>>> resultDataset.createIndividual(OT.OTClass.FeatureValue.getOntClass(resultDataset));
>>>                 featureValue2.addProperty(OT.feature, feature2);
>>>                 featureValue2.addLiteral(OT.value,
>>> resultDataset.createTypedLiteral(this.fingerprints[i],
>>> XSDDatatype.XSDdouble));
>>>
>>>                 //and finally add values to the data entry
>>>                 dataEntry.addProperty(OT.values, featureValue1);
>>>                 dataEntry.addProperty(OT.values, featureValue2);
>>>             }
>>> *******************************************************
>>>
>>>
>>
>> Can't really spot what's wrong, will try to run it.
>>>
>>> On 3 September 2010 22:34, Nina Jeliazkova <jeliazkova.nina at gmail.com>
>>> wrote:
>>> > Hi Surajit,
>>> >
>>> > I assume resultDataset variable is a Jena Model?
>>> >
>>> > Here is one of the RDF files you've tried to upload
>>> >
>>> > http://www.ideaconsult.net/downloads/maxtox/ambit2_22761.rdf
>>> >
>>> > First observation there is no any ot:Dataset instance. There are multile
>>> > ot:DataEntry, but no dataset to where they belong.  This is definitely a
>>> > reason the dataset could not be imported - the parser simply doesn't
>>> > find a
>>> > dataset to work with and does nothing.
>>> >
>>> > Second observation - you have defined value of type double, but the
>>> > value
>>> > itself is a string of 0 and 1s (the fingerprint).  This is at least not
>>> > correct and will most probably result in errors when parsing values.
>>> >
>>> >        <ot:FeatureValue>
>>> >             <ot:value
>>> > rdf:datatype="http://www.w3.org/2001/XMLSchema#double"
>>> >
>>>
>>> >> >>0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0</ot:value>
>>> >             <ot:feature
>>> > rdf:resource="http://localhost:8080/MaxtoxTest/feature/1_fingerprint"/>
>>> >           </ot:FeatureValue>
>>> >
>>> Corrected that ... I had interchanged the types between prediction and
>>> fingerprint
>>>
>>> > Ah, and (relatively minor)  issue - the feature URI
>>> > http://localhost:8080/MaxtoxTest/feature/1_fingerprint will not be
>>> > dereferencable, even if you are running feature service at your machine
>>> > -
>>> > just because "localhost" means your machine for you, but my machine to
>>> > me.
>>> >
>>>
>>> Yeah ... that was during testing from my machine ...I have loaded the
>>> same on the freiburg server which has the correct (absolute) server
>>> path.
>>>
>>> > If you are not running feature service elsewehere it would be better to
>>> > either have anonymous node for it, or create it beforehand  (POST to
>>> > http://{host}/ambit2/feature) and provide the URI in the dataset RDF.
>>> > TUM
>>> > and NTUA developers can share their experience/code as well.
>>> >
>>>
>>> Will try that in a future iteration ... however I have a thought
>>> regarding this. Remoting is good as long as the information load to be
>>> maintained is high and the connection is good. I believe localizing
>>> the smaller stuff is better - as it leads to fewer steps that can fail
>>> (in the course of a bad connection)
>>
>> Both approaches above a valid, the point is to have the information about
>> the feature stored somewhere and linked to the rest.
>> Low quality connection is what stateless protocols are good for.
>>
>>>
>>> > Third,  this  code will not work as you might expect:
>>> >
>>> > this.uploaded_URI = output.getText(); ,
>>> >
>>> > because task RDF will be returned, not a single URI (you have specified
>>> > request.getClientInfo().getAcceptedMediaTypes().add(new
>>> > Preference<MediaType>(MediaType.APPLICATION_RDF_XML));
>>> >
>>>
>>> Do I have to drill through the RDF representation to reach the URI ?
>>
>> If requesting RDF, yes.  But you might also set accepted media type to
>> text/uri-list , this will return URI only.
>>
>>>
>>> Also if its taking a long time what is your polling strategy ?
>>>
>>
>> The polling strategy is decided by the client side.
>>
>>>
>>> Are there any ready pieces of code for polling ?
>>>
>>>
>> Look at RemoteTask class at the opentox-client SVN link
>> Nina
>>
>>>
>>> > Hope this helps,
>>> > Nina
>>> >
>>>
>>> Lastly some links I had left out (for reference)
>>>
>>> models at :
>>> http://opentox2.informatik.uni-freiburg.de:8080/MaxtoxTest/model
>>> particular (for ex model no 1) :
>>> http://opentox2.informatik.uni-freiburg.de:8080/MaxtoxTest/model/1
>>>
>>> models take a dataset as the input and generate the fingerprints and
>>> the predictions.
>>>
>>> Regards
>>> Surajit
>>>
>>> >
>>> > On Fri, Sep 3, 2010 at 6:33 PM, surajit ray <mr.surajit.ray at gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Tried uploading using this piece of code
>>> >>
>>> >> ************************************
>>> >> System.out.println(result_dataset_URI);
>>> >>            Representation representation = new
>>> >> OutputRepresentation(MediaType.APPLICATION_RDF_XML) {
>>> >>
>>> >>                @Override
>>> >>                public void write(OutputStream out) throws IOException {
>>> >>                    resultDataset.write(out, "RDF/XML-ABBREV");
>>> >>                }
>>> >>            };
>>> >>
>>> >>            Request request = new Request();
>>> >>            Client client = new Client(Protocol.HTTP);
>>> >>            request.setResourceRef(result_dataset_URI);
>>> >>
>>> >>  request.setReferrerRef(MaxtoxApplicationSettings.getServerRootPath());
>>> >>            request.setMethod(Method.POST);
>>> >>            request.getClientInfo().getAcceptedMediaTypes().add(new
>>> >> Preference<MediaType>(MediaType.APPLICATION_RDF_XML));
>>> >>            request.setEntity(representation);
>>> >>
>>> >>
>>> >>            Response response = client.handle(request);
>>> >>            // Get the output from the response
>>> >>            Representation output = response.getEntity();
>>> >>            this.uploaded_URI = output.getText();
>>> >> ************************************
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> I got this as the result
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> *************************************
>>> >> 0.0 Error 1283527568408 File import application/rdf+xml [-1] The
>>> >> server has fulfilled the request but does not need to return an
>>> >> entity-body, and might want to return updated metainformation The
>>> >> server has fulfilled the request but does not need to return an
>>> >> entity-body, and might want to return updated metainformation 500
>>> >> *************************************
>>> >>
>>> >> Could you clarify ?
>>> >>
>>> >> I am using the http://apps.ideaconsult.net:8080/ambit2/dataset for
>>> >> testing as the other server was down - sorry for that.
>>> >
>>> >
>>> >>
>>> >> I just uploaded
>>> >> twice so you can delete 2 vague looking databases ...
>>> >
>>> >
>>> >
>>> >>
>>> >> Regards
>>> >> Surajit
>>> >>
>>> >>
>>> >>
>>> >> On 3 September 2010 11:46, Nina Jeliazkova <jeliazkova.nina at gmail.com>
>>> >> wrote:
>>> >> > Hi Surajit,
>>> >> >
>>> >> > On Fri, Sep 3, 2010 at 1:50 AM, surajit ray
>>> >> > <mr.surajit.ray at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> From your previous example I gathered this
>>> >> >>
>>> >> >> RDFRepresentation r = new RDFRepresentation();
>>> >> >> Representation rep = r.createRepresentation(rdfDataset,
>>> >> >> MediaType.APPLICATION_RDF_XML);
>>> >> >>
>>> >> >> The problem is this representation does not exist in the stable
>>> >> >> version of Restlets !
>>> >> >
>>> >> > If you refer to opentox-client examples, then RDFRepresentation is
>>> >> > not a
>>> >> > Restlet class at all, but custom one, using Jena writers for
>>> >> > serializing
>>> >> > RDF.
>>> >> >
>>> >> >
>>> >> >
>>> >> > https://ambit.svn.sourceforge.net/svnroot/ambit/branches/opentox/opentox-client/src/main/java/org/opentox/rdf/representation/RDFRepresentation.java
>>> >> >
>>> >> > Restlet RDF support is rather minimalistic (graph representation, no
>>> >> > querying, let alone reasoning), I am not sure if any of the partners
>>> >> > actually uses it. If you are working with 1.x Restlet , there is
>>> >> > indeed
>>> >> > no
>>> >> > RDF support there. But Restlet RDF support can be safely ignored if
>>> >> > you
>>> >> > are
>>> >> > already using Jena (though I would be curious to see performance
>>> >> > comparison
>>> >> > between the two).
>>> >> >
>>> >> > Using StringRepresentation/ FileRepresentation with media type
>>> >> > "application/rdf+xml" does the same job.  RDFRepresentation class
>>> >> > discussed
>>> >> > above actually creates OutputStreamRepresentation.
>>> >> >
>>> >> > Upload examples can be found here
>>> >> >
>>> >> >
>>> >> >
>>> >> > https://ambit.svn.sourceforge.net/svnroot/ambit/branches/opentox/opentox-client/src/test/java/org/opentox/dataset/test/DatasetTest.java
>>> >> >
>>> >> >
>>> >> > https://ambit.svn.sourceforge.net/svnroot/ambit/branches/opentox/opentox-client/src/test/java/org/opentox/dataset/test/FeatureTest.java
>>> >> >
>>> >> > The examples above are cleaned up versions of ambit code. JUnit tests
>>> >> > of
>>> >> > ambit dataset services can be found here
>>> >> >
>>> >> >
>>> >> >
>>> >> > https://ambit.svn.sourceforge.net/svnroot/ambit/trunk/ambit2-all/ambit2-www/src/test/java/ambit2/rest/test/dataset/DatasetsResourceTest.java
>>> >> >
>>> >> >
>>> >> >
>>> >> >>
>>> >> >> Also the example uses a RemoteTask to achieve the actual upload.
>>> >> >> Therefore I could not exactly fathom how to do the upload.
>>> >> >
>>> >> > Here is a summary
>>> >> >
>>> >> > curl -X POST -d "@filecontent" -H "Content-type:application/rdf+xml"
>>> >> > http://datasetservice/dataset
>>> >> >
>>> >> > This returns Task object (either URI, if -H "Accept:text/uri-list" is
>>> >> > specified or Task RDF representation if -H
>>> >> > "Accept:application/rdf+xml"
>>> >> > is
>>> >> > specified).  Task RDF representation is defined in
>>> >> > http://opentox.org/api/1_1/opentox.owl
>>> >> >
>>> >> > If the task is still running you'll get result codes 201 or 202 ,
>>> >> > this
>>> >> > means
>>> >> > you need to query again the returned task URI.  If code 200 is
>>> >> > returned,
>>> >> > the
>>> >> > URI returned is where the result should be found.   In the examples
>>> >> > there is
>>> >> > a RemoteTask class that handles all this.
>>> >> >
>>> >> >
>>> >> >
>>> >> > https://ambit.svn.sourceforge.net/svnroot/ambit/branches/opentox/opentox-client/src/main/java/org/opentox/task/RemoteTask.java
>>> >> >>
>>> >> >> On 3 September 2010 03:47, surajit ray <mr.surajit.ray at gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > I have managed to get a dataset (RDF) made using the results of a
>>> >> >> > model prediction run. Could you please tell me a dataset service
>>> >> >> > where
>>> >> >> > I could upload the same ?
>>> >> >
>>> >> > For testing
>>> >> >
>>> >> > http://ambit-uni-plovdiv.bg:8080/ambit2/dataset
>>> >> >
>>> >> > For your final models/results , especially if you would like them
>>> >> > appearing
>>> >> > in ToxPredict.
>>> >> >
>>> >> > http://apps.ideaconsult.net:8080/ambit2/dataset
>>> >> >
>>> >> >
>>> >> >>
>>> >> >> Also it would be very helpful if you could
>>> >> >> > give an example of an upload request using restlets ...
>>> >> >
>>> >> > See above.
>>> >> >
>>> >> > Hope this helps,
>>> >> > Nina
>>> >> >
>>> >> >
>>> >> >> >
>>> >> >> > Thanx
>>> >> >> > Surajit
>>> >> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >
>>>
>>> >
>>
>>
>>
>

-- 
Surajit Ray
Partner
www.rareindianart.com