SPARQL DELETE/INSERT with SERVICE statement

I'm creating a local cache for some data from Wikidata, to make sure I have fast autocompletion for tags. I would like to update the data once a week but only if the SERVICE clause works.

Basically I do:

INSERT { GRAPH <http://my.data/graph/wikidata> {
    ?concept wdt:P902 ?hls ;
        rdfs:label ?label ;
        schema:description ?description .
}} WHERE {
    SERVICE <https://query.wikidata.org/sparql> {
        ?concept wdt:P902 ?hls .
        ?concept rdfs:label ?label .
        ?concept schema:description ?description .
        FILTER (lang(?description) = "en")
        FILTER (lang(?label) = "en" || lang(?label) = "de" || lang(?label) = "fr" || lang(?label) = "it")
}}

Now I thought I can do a DELETE/INSERT and I delete all data first:

PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX schema: <http://schema.org/>

WITH <http://my.data/graph/wikidata>
DELETE { ?s ?p ?o }
INSERT {
    ?concept wdt:P902 ?hls ;
        rdfs:label ?label ;
        schema:description ?description .
} WHERE {
    SERVICE <https://query.wikidata.org/sparql> {
        ?concept wdt:P902 ?hls .
        ?concept rdfs:label ?label .
        ?concept schema:description ?description .
        FILTER (lang(?description) = "en")
        FILTER (lang(?label) = "en" || lang(?label) = "de" || lang(?label) = "fr" || lang(?label) = "it")
    }
    ?s ?p ?o
}

But like this it looks like I get 400 back from Wikidata. I do similar DELETE/INSERTs without SERVICE but I can't see why this would not work like this, as I do not bind any variables between the existing graph and the SERVICE query.

Anyone sees what's going wrong here? Basically I would like to wipe the existing graph but not end up with an empty graph in case Wikidata is down.

Thanks to user AKSW for the hint with the UNION, this is the query that works:

PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX schema: <http://schema.org/>


DELETE { GRAPH  <http://data.alod.ch/graph/wikidata> { ?s ?p ?o }}
INSERT { GRAPH  <http://data.alod.ch/graph/wikidata> {
    ?concept wdt:P902 ?hls ;
        rdfs:label ?label ;
        schema:description ?description .
}} WHERE {
{
    SERVICE <https://query.wikidata.org/sparql> {
        ?concept wdt:P902 ?hls .
        ?concept rdfs:label ?label .
        ?concept schema:description ?description .
        FILTER (lang(?description) = "en")
        FILTER (lang(?label) = "en" || lang(?label) = "de" || lang(?label) = "fr" || lang(?label) = "it")
    }
}
UNION
{
    GRAPH  <http://data.alod.ch/graph/wikidata> {
      ?s ?p ?o
    }
}
}