***
Wartungsfenster jeden ersten Mittwoch vormittag im Monat
***
Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
VRE-Repository-Connector
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Requirements
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Deploy
Releases
Package registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Center for Research Data Management
VRE-Repository-Connector
Commits
8ab488c0
Commit
8ab488c0
authored
1 year ago
by
Moser, Maximilian
Browse files
Options
Downloads
Patches
Plain Diff
Implement `auto.download()` of all files for a container
parent
efba3713
Branches
Branches containing commit
Tags
Tags containing commit
1 merge request
!5
v0.2.0
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
vre_repository_connector/auto.py
+38
-15
38 additions, 15 deletions
vre_repository_connector/auto.py
with
38 additions
and
15 deletions
vre_repository_connector/auto.py
+
38
−
15
View file @
8ab488c0
...
...
@@ -5,7 +5,7 @@
"""
Helpers for interacting automatically with various repository types.
"""
from
typing
import
Optional
,
Tuple
from
typing
import
List
,
Optional
,
Tuple
from
urllib.parse
import
urlparse
import
pandas
as
pd
...
...
@@ -74,38 +74,61 @@ def suggest_repository(url: str) -> Optional[BaseWrapper]:
if
service_cls
is
None
:
return
None
return
service_cls
(
url
)
return
service_cls
(
f
"
{
scheme
}
://
{
host
}
"
)
def
download
(
url
:
str
,
all
:
bool
=
False
,
interactive
:
bool
=
True
)
->
Optional
[
str
]:
def
_download
(
service
:
BaseWrapper
,
container_id
:
str
,
files
:
List
[
str
]
|
str
,
interactive
:
bool
)
->
List
[
str
]
|
str
:
"""
Download one or more files.
"""
results
=
[]
already_auth
,
single_file
=
False
,
False
if
isinstance
(
files
,
(
str
,
int
)):
files
,
single_file
=
[
files
],
True
for
file
in
files
:
try
:
dl
=
service
.
download
(
container_id
,
file
)
results
.
append
(
dl
)
except
Exception
as
e
:
if
interactive
and
not
already_auth
:
service
.
authenticate_interactive
()
already_auth
=
True
dl
=
service
.
download
(
container_id
,
file
)
results
.
append
(
dl
)
else
:
raise
e
return
results
[
0
]
if
single_file
else
results
def
download
(
url
:
str
,
all
:
bool
=
False
,
interactive
:
bool
=
True
)
->
List
[
str
]
|
str
|
None
:
"""
Download file automatically based on the URL.
"""
if
(
service
:
=
suggest_repository
(
url
))
is
None
:
return
None
# fish out the container & file from the URL
cid
,
fid
=
service
.
url_to_parts
(
url
)
cid
,
fid
,
fids
=
*
service
.
url_to_parts
(
url
)
,
[]
if
cid
is
None
and
fid
is
None
:
cid
,
fid
=
service
.
url_to_parts
(
_follow_redirects
(
url
))
# if we couldn't determine a file
, ask the user in interactive mode
# if we couldn't determine a file
from the URL...
if
fid
is
None
:
if
all
:
#
TODO get a list of all files
pass
#
we either take all of them
fids
=
list
(
service
.
list_files
(
cid
))
elif
interactive
:
#
TODO
ask the user
which file to download
#
or we
ask the user
(TODO)
pass
else
:
# or we give up
return
None
try
:
return
service
.
download
(
cid
,
fid
)
except
Exception
:
# if the download didn't work, we can try again with authentication
if
interactive
:
service
.
authenticate_interactive
()
return
service
.
download
(
cid
,
fid
)
return
_download
(
service
,
cid
,
fid
or
fids
,
interactive
)
finally
:
service
.
clear_auth
()
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment